Date Filtering Macro

[sjm]

Hi - I'm looking to scrape several URLs but I'm finding that it's too much data to pull, even with throttle tools. My workflow will run quickly if I manually add a date filter to just download data for URLs associated with a particular date.

Any ideas on how to create a macro to filter for one date at a time, then download the relevant pages, do transformations, write to a database AND THEN start all over again with the NEXT date?

The incoming data into the webscraping download tool is simply:

- date of event

- event URL 

Everything in my workflow gets me to the desired results... I just need guidance on running the process for one date at a time so the workflow doesn't get overloaded.

Thanks! 

[binuacs · Accepted solution]

@sjm create a batch macro and pass the date to the control parameter and call the url inside the macro

[sjm]

Thanks @binuacs for the quick reply. I just tested your macro for a small # of dates and it worked. I'm going to see if this will work for a few hundred dates and thousands of URLs.

Thank you!