Iterative API call with pagination & updating URL

nakamott

I am struggling to come up with a macro that iterates on the condition that I have returned 50 records.

I found this post which is similar to what I want:

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Iterative-macro-for-API-pagination-retrieval/m-p/34456#M13729

I do not have any indicators though that there are more records to retrieve. I just have to assume there are more records if the previous call returned 50. I do not want to stop until the call returns < 50 records.

I want to iterate this endpoint and increase offset by 50 each time:

analyst.onclusive.com/api/v1/profiles/279/content_items?from=2019-08-06&to=2019-08-07&offset=0

analyst.onclusive.com/api/v1/profiles/279/content_items?from=2019-08-06&to=2019-08-07&offset=50

analyst.onclusive.com/api/v1/profiles/279/content_items?from=2019-08-06&to=2019-08-07&offset=150

Then I want to stop iterating the first time less than 50 records are returned. With each call I also want to append the records that I returned. 

So how do I update the URL with each iteration?

And how do I append records with each pass?

And how do I check if less than 50 records were returned?

Here is what I have so far:

DiganP

@nakamott You would have to use the iteration macro. You have to use the Interface Designer to set the loop. Part of the look will be the results and the 2nd part would be to create a string of the next url. Attached is a sample workflow that I like to show to clients around this use case. 

Web Scraiping with Pagination.yxzp