Iterative Macro to loop API calls

Question

Hi!

Fairly new Alteryx user here. I'm trying to pull historical messages from the Yammer API using a macro. The API limits its response to 20 messages at a time but the older_than query parameter will let you pull 20 older messages by adding the message ID of the oldest original message in a new API call. The macro is supposed to do this loop until a certain date and union the output. I've tried a couple of ideas but have hit a roadblock.

1. My original macro design was not dynamic and I hard set it to stop after 65 iterations knowing the 64th was the last to receive data. My loop filter would not stop it before then. My thought is that the append fields would output even if one anchor did not receive data sending it into an infinite loop. However, the full data set would union and appear in the main workflow.

2. The new design idea was to pull the channel creation date from the meta info in the original API call in the main workflow and pass it into the macro as "validation date", put the append tool before the filter, and filter off whether the min date was > validation date. Then I would add a date and action interface tool in the main workflow to update the validation date and make the macro fully dynamic. The new macro design runs without error in isolation but does not output to the main workflow.

Any ideas why the output might not be unioning? Is an iterative macro the right idea? Does the engine iteration number have to be used?

Thanks!

Felipe_Ribeir0 · Answer

@AustinSalter1 Ok, if your stop condition is working fine, i understand that with the bellow configuration for your macro the output should come.

If the provided answer helped you to solve the problem/was correct, please accept it as a solution :)

Thanks.

AustinSalter1 · Answer

Felipe,

Thanks for your help! The no data idea is good but I would prefer to be able to dynamically stop the macro with the date (Stop condition: min date> date input). The yxmc workflow produces data.

Felipe_Ribeir0 · Answer

Hi @AustinSalter1

Iterative macros for me are the best way to solve this problem, since they work like the for/while commands of programming languages, which are normally the commands to solve this pagination problem.

If i understood well, what is your main problem is that you are not giving the macro the "stop condition" of the loop. Hard coding a number of iterations is not the best way, since the necessary number of iterations usually increase over time (because there is always more data).

1)I solved a similar problem with the Alchemer/Surveygizmo API (SurveyGizmo/Alchemer API - GET survey data - Alteryx Community). Take a look at the macro inside this workflow. I noticed that after the last iteration with data (for you 64), i was not getting .data fields (these were the fields that i was getting until the last valid iteration). So i filtered this and used the append tool to stop the workflow when there is no .data fields. This was my "stop condition".

2)With CVENT API (CVENT API - Alteryx Community), it was more or less the same. I noticed that after the last iteration with data, i was not receiving the next page token. So just filtering this field was enough to configure my "stop condition"

So, take a look at what is the response of the API on the 65 call and see how you can use this response with filters/append to set the stop condition of your workflow.

About the data that is not unioning, when you run the macro alone (just one iteration using the yxmc workflow) you are seeing data on the outputs?