I’m scraping a massive e-com catalog and downloading thousands of product images to my local drive. Right now, when the scraper finishes, I have to manually zip the folder and upload it to my AWS S3 bucket.
Is there a way to automate this post-run cleanup within RTILA?
Yes! You can chain multiple triggers together to create a completely hands-off pipeline.
First, add a File Formats trigger and select the compress_file operation. Point it to your downloaded images folder and set the format to zip.
Next, add an AWS S3 trigger right below it. Set the operation to upload. For the source_path, you can dynamically reference the output of your previous compression step using a variable. The Python Relay will execute these sequentially: the scraping engine finishes, the relay zips the files, and then it natively uploads the zip to S3 using boto3 under the hood. No third-party automation tools required!