I’m scraping a modern SPA that returns some really complex, deeply nested JSON data in the DOM. Relational databases are a nightmare for this, so I want to dump the raw records straight into MongoDB.
What’s the syntax for passing an entire scraped record into the Mongo trigger?
MongoDB is perfect for this kind of unstructured data!
In your MongoDB trigger, set the operation to insert_one. In the document field, instead of mapping out individual variables like {title} or {price}, you can just pass the entire dataset record using the root variable: {dataset}.
If your project is set to iterate mode, the Python Relay will loop through every scraped item and insert it as a distinct document into your Mongo collection. Our relay maintains a persistent connection pool (MongoClient) during the loop, so it can blast through thousands of inserts in seconds without dropping connections.