Iterate Download tool based on the output

[KarteekChalagondla]

Hi All,

I have to make a HTTP request using the Download tool to get 999 records in Json format. Along with the JSON output there will a Nextlink (another Http link) to get the next 999 records and the process goes on untill the Nextlink is empty. So I have to make a iteration at Download tool based on the output data.

How can I achieve this? Anyone faced similar situation?

[Jonathan-Sherman · Accepted solution]

So you will need to build an iterative macro to solve this. Create your standard workflow (with a macro input) and filter at the end to separate out "next link" and "JSON data". One macro output tool after each and in the interface designer select the macro input, iterative output will be your macro output from output anchor giving you the "next link". You will need to ensure you have selected iterative macro in the workflow options tab of the configuration pane to give you the correct options in the interface designer. This will "loop" the workflow and use the output of the "next link" stream as the input once again, through the download tool etc and finish running when no further data comes out of your iterative macro output "next link". Assuming you don't hit the maximum number of iterations defined in the interface designer, which can be changed.

If this solves your issue please mark the answer as correct, let me know if you need me to build an example!

Jonathan

[afv2688]

Hello @KarteekChalagondla,

It is possible to perform what you are looking for, you just need to create an iterative macro for that.

Remember that when creating iterative macros, iterative input should be the same as the iterative output.

Regards

[KarteekChalagondla]

Hi @Jonathan-Sherman 

Thank you very much for the solution. It works. But what If I set iteration limit to 100000 and the NextLink is blank at may be 250 or 300. Will it continue to loop, or throws exception? 

[Jonathan-Sherman]

The iterative macro will finish running when no data comes through the defined iterative output as there is no value to put back into the macro input. As @afv2688 said, you also need to ensure the schema is the same between iterative macro output and macro input.

[KarteekChalagondla]

Hello @afv2688 ,

Thank you very much. It helps alot. My first few trails failed due to the wrong assignements.