Iterate the Download Tool with multiple inputs.

[swadson]

I am trying to have the download tool make API calls one at a time rather than doing all parallel runs as I have multiple inputs coming into the download tool for different date parameters.  Since the download tool makes parallel calls my API end points are timing out. Is there a way to control that way to make the api calls one at a time?

[apathetichell]

I am 95% sure the download tool does make calls one at a time. your issue is that they are still too fast for your timeout and you're sending in too many records at once. Use a batch macro. add the tool to the batch macro. send in one record at a time. use python or @MarqueeCrew 's tool to pause your workflow.

[SPetrie]

To add to what @apathetichell mentioned, there is also a throttle tool in the designer tools that will help limit the number or records that are processed per minute. That will also help limit how many requests go through at a time.

[TimN · Accepted solution]

Also, there are options to throttle within the download tool itself under the connection tab.

[swadson]

@SPetrie  -  In my case I dont have an issue processing the number of records. Its more at the API End Point that timing out. We did as the API Owner to increase the time out though. But if the download tool is going to make one call at a time, then I should be good,  Thank you!

[swadson]

@TimN  - Thank you! I had it set at 4 connections as I was make close to 45 Calls.  I was able to schedule the flow and it kept running for about 2 weeks. This was causing a lot on the EndPoint and seems like something changed and it started failing after.

I can go with 1 or 2 active connections, but the issue is that as we go forward the number of calls that we will have to make will have to increase from 45 to add more time segments. This might make the flow to run for a longer time frame. 

I'll test it out with just 2 connections and hope it works.

[TimN]

Hi,

If I helped, please like and mark as the solution.

Thanks,

Tim