Iterative Macro with API pagination

Hello, I have an API that I pass a limit and an offset.The first pass I have a constructed URL string that starts at offset=0

The first pass returns a JSON Value for a property called "nextPageLink" which already has the next offset.

From my workflow, I get an authorization bearer token and pass that to the macro call.

Inside the macro, the first tool I have is a formula that sets the initial URL value.

I am getting the results and splitting everything except the "nextPageLink" to the output. I send "nextPageLink" to the Loop.

On iteration 1, it works fine. On Iteration 2 I am getting error: RPSMacro (63) Iteration #2: Tool #31: The field schema for the output "Loop" changed between iterations.

I am so stuck my eyes are crossed.

Macros

API

Accepted answers

gabrielvilella

The data that is being returned to the input is the one you have defined as the iteration output within interface designer.

All comments

gabrielvilella

To get rid of the schema error you need to just select the Auto Configure by Name within Interface Designer > Properties area.

brwright

Thanks, I had done that but I did it from the standard workflow instead of the macro workflow. (This wasnt clear in the help article I initially read). That schema message has gone away. Now I have the following I'm staring at with zero percent.

I set the macro's max iterations to 2 so it would stop and I could take a look at whats being returned. Seems to be in an infinite loop regardless of that setting. If I stop it, I cant see any output.

brwright

2 iterations ran. I have the max iteration at 2. I dont understand why it's still going.

gabrielvilella

8537 iterative records remaining is just the record count for the iterative output anchor. It means that you still have records going through the iterative output. The iterative macro will only stop when there are no more records left on the iterative output or the maximum number of iterations is reached.

brwright

I have to send an Authorization token and the nextpageLink. How do I validate "what" is getting sent each round? I dont understand the relationship between the loop and where it replaces the url and gives the download the Authorization header payload.

gabrielvilella

The data that is being returned to the input is the one you have defined as the iteration output within interface designer.

brwright

The Input only has Authorization. So I thought maybe I need to include a URL too. The 2nd run was "unauthorized" . Now I'm getting new error that URL was missing. What a battle.

gabrielvilella

Nice!

brwright

I just studied my results and they are duplicates of the first call. It's not feeding the nextURL. I've opened a support ticket. I dont understand how the output contains a value and somewhere upstream should grab that value and pass it it through the iteration.

jrobiso2

You need 2 inputs. One is the URL, and the other is the AUTH. Separate them, because it has to be designated as the Iterative Input. Your Iterative macro will need the URL input to be designated as the Iterative INPUT, and the i output designated as iterative output. Now the fun part - you'll need to parse the metadata from the download to find the nextLevel value, and that goes to your iterative output.

In the overall macro config, settings section, is where you tell it which input/outputs are the iterative ones.

Quick Links

This months top contributors

atcodedog05 19598

Qiu 15867

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703