Iterative API call with pagination & updating URL

[nakamott]

I am struggling to come up with a macro that iterates on the condition that I have returned 50 records.

I found this post which is similar to what I want:

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Iterative-macro-for-API-pagination-ret...

I do not have any indicators though that there are more records to retrieve. I just have to assume there are more records if the previous call returned 50. I do not want to stop until the call returns < 50 records.

I want to iterate this endpoint and increase offset by 50 each time:

analyst.onclusive.com/api/v1/profiles/279/content_items?from=2019-08-06&to=2019-08-07&offset=0

analyst.onclusive.com/api/v1/profiles/279/content_items?from=2019-08-06&to=2019-08-07&offset=50

analyst.onclusive.com/api/v1/profiles/279/content_items?from=2019-08-06&to=2019-08-07&offset=150

Then I want to stop iterating the first time less than 50 records are returned. With each call I also want to append the records that I returned. 

So how do I update the URL with each iteration?

And how do I append records with each pass?

And how do I check if less than 50 records were returned?

Here is what I have so far:

[DiganP · Accepted solution]

@nakamott You would have to use the iteration macro. You have to use the Interface Designer to set the loop. Part of the look will be the results and the 2nd part would be to create a string of the next url. Attached is a sample workflow that I like to show to clients around this use case. 

[GreggD]

Thanks for your reply. I have tried locating the "Iterative Macro" both in the tool and in the search bar of the tool without success. FYI - I could not load your workflow because it was developed with a newer version than I have. I am operating with 2018. If you could point me to where I can find the "Iterative Macro" that would be a start. Thanks.

[BretCarr]

Your example is fantastic. It's the first example I've seen that I've actually understood. Thank you!

[BretCarr]

Here is the link to my visual representation of the process using Alteryx iteration macro!

https://community.alteryx.com/t5/Welcome-Introductions/Need-to-build-Iteration-Macro-to-paginate-API...

Good luck! These things are a pain and take a lot of "getting your head around" them. They can also be touchy with the input types/values.

[Dennisklad1]

Hi, thanks for the reply.

I have a similar problem where I want to paginate my request per 300 request. I though the best way to go through all my results is to break the loop when the returned records are less than 300, meaning its the last page.

This is my macro, but I am not sure if its working correctly. I want per iteration the field parameter to increase by 300 and then as update the input url.