30 million instrument universe - URL link data source - How can I pull data in stages?

Question

Hi, I want to pull a very large data set of 30Million lines of data which I have a key identifier and pull using an URL. This works for smaller universes, but I need to change my workflow to pull the data in stages and then join together? I normally pull JSON files.

Can anyone help?

Thanks

Leigh

OTrieger · Answer

@Bmthred1980

Hi Leigh,

You should try to set batches, let say a batches of 50,000 lines at a time by setting ranges. Then run it with Batch Macro, it will each time pull in 50,000 lines and at the macro output you will have all the data union. Will you be able to set ranges for your data?