We are building a real-time competitor price monitoring dashboard. We need the scraper to push pricing updates directly into our Apache Kafka cluster the moment they are extracted.
Does RTILA support streaming data to Kafka, or do I need to write a custom script to watch the output files?
You don’t need to write any custom scripts. RTILA X has a native Kafka trigger built right into the Python Relay.
When you configure the trigger, you just provide your bootstrap_servers and the topic name. The real magic is in how you configure the dataset mode. If you set the trigger to run in iterate mode, the relay will fire a message to your Kafka topic for every single row as it processes the dataset.
Just set your message payload to something like:
{"product": "{name}", "price": "{price}"}
Because the Python sidecar runs locally alongside the scraping engine, the latency is near zero. It’s a true enterprise-grade pipeline right out of the box.