Scrape Multiple URLs with Batch Macro

[pgensler]

Hello,

I am trying to create a batch macro where I can iterate over multiple URL's and extract a brewery name, like the workflow attached. What type of macro would work best for this, and do I have my workflow set up to increment? It seems like it works to get what I want from the website, but I'm not sure how to save the results after the first row has executed to move onto the next URL. Any help would be appreciated. Thanks.

[andre347 · Accepted solution]

Hi Peter,

I've attached two workflows. The first one is a batch macro that downloads the data. It's just your workflow but then with a control parameter on the URL field. The second workflow is a normal workflow that has a list of urls. You use this field to loop through the batch marco. So the URLs in this workflow will run through your batch macro and everything will be unioned.

You would want to include the data cleaning part of the workflow within the batch macro.

Let me know if you need more help.

[pgensler]

I have made some minor tweaks to the workflow, but I'm getting an error where the field schema changed between iterations?? Is there any easy way to fix this? Workflow is attached.

[andre347]

Yes, go to the Interface Designer (under View) and then change the last setting to 'Auto Configure by Name'.