Batch Macro Control input from within Macro itself

[jrobiso2]

I have an issue where I am pulling from an API that will only return 100 "items" of JSON at a time (in this case, each is info on a machine).

As part of the returned metadata, there is a "nextLink" value which is the url to use to get the next 100.

In previous API work, I've known how many are returned due to initial metadata (usually total=xxxxx) and known page size, and I could use the 1st call to create a Control input table like "page=1, page=2, page=3" (total/page size) etc. But this time, there is no metadata on total items, nor a paging function. Merely a "nextLink" to call to get the next 100 items.

Is there any way to incorporate that returned metadata "nextLevel" into the control param of a batch macro so that it would change the URL field value?  I would also need  to test the "nextLevel" metadata for blank, meaning I've already processed all the items.

[JamesCharnley]

@jrobiso2 instead of a batch macro, you'll want to use an iterative macro in this instance to run the workflow again. You'll want to isolate that NextLink value as part of your iteration output, so next iteration the macro will instead call the next page, and the process repeats for as many pages as you need. The 100 rows at a time can be sent out of a different output each iteration.

[Felipe_Ribeir0]

Hi @jrobiso2 

Probably it will be a little hard to understand what is happening without access to the data, but in these 2 iterative macros (for SurveyGizmo and Cvent systems) i used a similar idea to solve this, maybe you can replicate the idea. 

SurveyGizmo iterative macro:

nextpage = currentpage + 1

when we reach the last page + 1, the filter tool + sample tool + append tool will not allow data to feed the next url (because after the last page, the filter will not allow data to pass through), so the process stop

https://community.alteryx.com/t5/Community-Gallery/SurveyGizmo-Alchemer-API-GET-survey-data/ta-p/999...

Cvent iterative macro:

filter next page url from current iteration

when we reach the last page + 1, there is no data to feed the next url (the Cvent system itself do not send the next page url if there is no data on the next page), so the process stop

https://community.alteryx.com/t5/Community-Gallery/CVENT-API/ta-p/998955

[jrobiso2]

Excellent. I'm working on an iterative macro now.

Hard to determine stop point, as I don't know what the final page metadata looks like for graph.microft.com/v1.0/devices - and it keeps erroring on bad http after about 1200 iterations.  Adding an R stop timer to slow down the iterations - API's don't like being slammed so fast!

[Felipe_Ribeir0]

So 1200 is probably close or after the end of the data i believe. You could try to run the macro alone with 1100, 1200 to see it.

[jrobiso2]

Roughly 250,000 devices, 100 at a time . . . so 2500 iterations is what I expect.  The final iteration where it crashed had a valid nextLevel metadata tag, which I verified using Postman, so I think it's definitely a timing issue.  I've had it before with API's when you send call after call with no timing between.

This is for graph.microsoft.com/v1.0/devices, enterprise-based.  

All this moving to API's vs good old-fashioned Database SQL is really turning out to be NOT what it's cracked up to be!

[JamesCharnley]

@jrobiso2 you might already know this but just in case you don't there are tools within Alteryx you can use to add some timing in between. The Throttle tool and the Wait a Second CreW macro come to mind as ways that may be applicable to slow down your calls!