Automating API Pagination in Alteryx – Seeking Suggestions

[Loperal]

Hello Everyone,

I've been working with Alteryx for a few years now, although not on a very intensive basis. Recently, I've encountered a challenge with a workflow that started out simple—just a few parameter changes—but has grown in complexity and now requires more frequent maintenance. This has led to occasional crashes and inefficiencies in the Designer.

To improve this, I'm looking to automate part of the workflow related to an API call. Specifically, I use the Formula Tool to construct the URL and handle pagination.

The Challenge

The API I'm working with only allows downloading 1,000 records per page. So, in Picture 1, I manually set the URL to retrieve the first 1,000 records and specify page 1. Then, in Picture 2, I skip the first 1,000 records and retrieve the next batch by setting page 2, and so on—this continues for each subsequent page.

While I know this can be solved using a macro or another automation method, there's one key requirement: I want to control how many records and pages are downloaded each time I run the workflow.

I've attached my workflow for reference. Any suggestions or best practices on how to approach this? Especially around building a dynamic pagination loop or using macros effectively?

Thanks in advance for your help!

 

[KGT · Accepted solution]

For starters, thank you for laying out the problem with examples etc.

You can construct a list using either generate rows or just a text input for all the skip numbers and then append that to your call and one formula will add the skip amount in, so this workflow would then break down to the below with this formula in your Formula Tool: [FULL_URL] = [BASIC_URL]+[GET_URL]+'?top=1000&skip=' + ToString([RowCount]). You can then control the number by controlling in between the Generate Rows and Append. Make sure Append is set to "Allow All Appends".

Generate Rows is starting on 0 and adding 1000 per loop.

As for Iterative Macro, you can find a few on the community to try and investigate/amend, and you can control stopping early, by using the iteration count and a filter, or testing something that relates to a variable entered into the macro. But I think you should be right for now with the above.

[Loperal]

Update – Feeling Grateful and Inspired

Just a few hours ago, I was stuck with an issue—and now I have a working solution I can build on. That feeling of satisfaction is truly rewarding!

I also wanted to take a moment to give back. Alteryx is a powerful tool, but what really makes it great is the community behind it. The willingness to share, support, and collaborate is what keeps us all growing.

 @KGT   many thanks for the Inspiration, this makes the difference

my workflow move from this

to this

 

 

[KGT]

Awesome work!! It really is rewarding, there's a reason that solving was the mantra for a couple of years!

I used to have a line that I mentioned, where Alteryx makes so many LOB data tasks so approachable, quick and possible that often someone takes a step early on and then takes many more steps without looking back to see if the first one was the best option now that they know more. As long as it works, it's better than before and so stays that way. 

Quite often people talk about "time saved", but IMO the benefit is what one can do that they didn't have time for or didn't have the right tool to approach an issue.