community
cancel
Showing results for 
Search instead for 
Did you mean: 

Alteryx designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.

Iterative macro for API pagination with multiple API calls

Fireball

Hello,

 

I have an iterative macro for API pagination.  My input contains multiple API calls.  I'm wondering if it's possible to run the iterative macro one at a time.  In other words, I would like to run one API call with pagination, then when there is no more pagination, run the next API call with pagination...

 

The issue I'm having with the iterative macro is I'm reaching the API rate limit although I have a throttle tool before the download tool.  Another issue is that some of the API calls didn't run successfully.

 

I believe those issues are due to the fact that I have API pagination with multiple API calls.  I believe the issue will be solved if I combine the iterative macro and the batch macro.  But I'm not succeeded it yet. 

 

Does anyone know how to run the workflow one at a time with pagination?

 

Sincerely,

Kazumi

Alteryx Certified Partner
Alteryx Certified Partner
Here's how I handle this:

Step 1 is find out how many pages will be called.
Step 2 is use the generate rows tool to create the right number of urls.
Step 3 is to then download them.

Cheers,

Mark
Alteryx ACE & Top Community Contributor

Chaos reigns within. Repent, reflect and reboot. Order shall return.
Fireball

Hi @MarqueeCrew,

 

Thank you for your reply.  I agree that creating all URLs before downloading them is a good idea for pagination.  After your reply, I dig into the API documentation.  I realized that there is a limit of 10,000 mentions that can be retrieved using the pagination (Link).

Pagination.png

So, I need to use maximumIdInResult,which is the latest id of a mention in the result, and use it for the next API call to retrieve all mentions (Link).

maximumIdInResult field.png

I think inserting the iterative macro within the batch macro would solve this issue.  I'm testing the batch macro right now.  I will let you know how it goes.

 

Sincerely,

knozawa

Alteryx Certified Partner
Alteryx Certified Partner

Hi @knozawa!

 

As you've allouded too, it's not always possible to get a list of all url pages before downloading because their is a depdency on the previous download to generate the 'link' if you will.

 

But you seem to have got to the solution which is to wrap your iterative macro within a batch macro.

 

I've attached an example of the same task but using the twitter API which you can perhaps use as a guide.

 

Ben

Alteryx Partner

Just jumping in to point out that if one of your API call returns a error in your Iterative Macro, it's maybe because of your API respons parsing inside that Macro.

 

Soma page returns a different structure than others and can't be output.

 

It may not be the case here but that's a possibility. 

A screenshot of your error could help determine that 

Alteryx Partner

Hi Ben

 

Has your solution been removed? I'm trying to run the packaged workflow, but without success. Any tips? Thank you

 

/Anders

Labels