I’m trying to scrape a catalog of about 50k e-com products. My issue is my ISP is a bit flaky, and if my connection drops at product 35,000, my standard Playwright Node script crashes and I lose hours of work. I have to start from scratch.
Does RTILA X have a way to handle network interruptions or pause/resume long-running jobs? Really don’t want to code my own queue system.
This is a classic headache with standard Playwright scripts lol. Yes, RTILA X handles this automatically through the built-in Checkpoint Manager.
Basically, as the Deno engine processes your URLs, it constantly syncs its state back to the local database sessions. It tracks exactly which URLs succeeded and which failed.
If your internet drops and the engine crashes, the Rust executor catches the non-zero exit code and marks the job as failed. When your internet comes back, you just hit “Retry Failed” in the UI. The engine reads the checkpoint, skips the 35,000 successful pages, and picks up exactly at page 35,001.