Macro feeds to sample tool

[MatthewHall]

Hi all, would appreciate some help as i am struggling to find a solution that works in Macros.

I have a geospatial dataset of 11,000 records that i need to run through a drivetime catchment API. The API is throttled to about 100 records at a time before i get timed out. So i need to be able to run the dataset in batches of 100 that doesnt start the next one until the previous is complete.

I can do this manually through using two sample tools, but each run takes 30mins and i don't have the time to sit all day and night re-running it. I cant find a way to feed the iteration into a sample tool in either batch or iterative macros.

I am happy to figure out the details on my own, but can someone point me in the right direction as i am struggling to figure out which tool will be best?  

[mceleavey]

Hi @MatthewHall ,

try creating a column that gives the records a number every 100 records (I suggest using the tile tool with equal split records of 100), then wrap the API call into a batch macro and group this record number to use as the control parameter value.

What this effectively does is tells the batch macro to go and run it in batches of 100, regardless of how long it takes to run that batch. This will not run the next one until the first batch of 100 has completed.

Before the macro output you could use the "Wait a Second" macro from the Crew macro pack to delay each batch by a second, for example.

This will achieve what you're after.

Hope this helps,

M

[MatthewHall]

many thanks, but i think what you are describing is a bit beyond my limited understanding of macro structuring! how do i get the macro to select the control parameter? not sure what tool i use to attach the tile selection to.

[mceleavey · Accepted solution]

Hi @MatthewHall ,

no problem. I've attached an example of how a batch macro works.

I've first created a Record ID, then re-set this number very 100 rows. Then I create a group number field, which is the first 100 records are group 1, the second 100 records are group 2 and so on.

I then grouped this field to get the unique number of groups.

I simply wrap the API call into a macro and tell it run in batches one run per group, so once for every 100 records in sequence.

The workflow looks like this:

You can right click the macro and add your API call to the box in the macro:

I hope this helps.

I would strongly recommend going through the macro training courses in the Academy section.

I hope this gets you started, anyway 🙂

M.

[MatthewHall · Accepted solution]

many thanks. After i sent my message, it suddenly dawned on my that i could use a filter tool and mostly got there. sadly i am running a slightly older version and i am unable to open your workflow, but should be able to figure it out.

I have been going through youtube videos and the lessons, but i find they all follow similar examples that are not really useful to me. I am usually good at self teaching this stuff, but this one has been stumping me.