Running Download tools in parallel

Question

Hi all,

I'm currently trying to speed up the process of downloading from an API. I've already moved from an iterative macro (changing the page number on each loop) to feeding multiple URLs (generated rows with iterative page numbers) to the download tools and that's given me a 10x performance increase, which is amazing!

But being a fiend for shaving off more time, I then had a cunning plan: To split those multiple URLs using the Tile tool and run them in parallel. That's given me something that looks like the below, but what I've noticed is that all the Download tools run sequentially, not in parallel! Weirdly, it starts from the bottom Download tool and moves up. So it saves no time. Is there something I'm missing here to get them to run at the same time?

dreldrel · Answer

If your API server won't flag you for abusing the API request, you can try a batch macro instead of an interactive macro. However, you would still need to use it cautiously, as most API servers nowadays have a limitation on the number of requests within a certain time period.

apathetichell · Answer

This is controlled by the maximum_connections in the connection part of the download tool and scales up to 32. Having different download tools (ie your tile tool method) probably doesn't matter ---> the maximum_connections setting does. If you want to use a batch macro---> and know your timeout messages (ie 429) ---> you can set this up with a batch macro/iterative macro to use a back off retry methodologyl