Throttling Google API Requests - Tried Throttle Tool and Crew Wait a Second Tools

[Scott_Snowman]

This question is based on Designer 2019.2

I am trying to design a workflow that sends batches of API requests to a Google API, but incorporates throttling so as not to exhaust quota limits.

I started with a single test case: bringing in a set of values representing the parameters for the API call, forming the URI for the call, requesting authentication via OAuth, and sending the whole package through the Download tool. Success, I got a JSON with results.

Keeping things light, I increased my input set to contain 4 records, with varying parameters, then repeat the process, I get 4 different JSON objects out. So far, so good.

The issue occurs when I try to throttle presentation of these 4 requests to the API via the Download tool. If I use either the Throttle tool or the CreW Wait a Second macro, I can tell that the records in the workflow are throttled before the Download tool, but it looks like the tool waits until all records are present before running through the Tool (which defeats the purpose).

See screenshot #1: 3 of the 4 records have passed through the Wait a Second Tool but the Download tool hasn't done anything yet. While the whole workflow takes 4 times the "wait a second" parameter, it looks like there are no API calls being done until the end of the run when they all happen sequentially with no pause.

I searched more and in this thread, on the advice of @cgoodman3 , I wrapped the API call via the Download tool in a Batch Macro. To get more visibility into the inner workings of the macro I had it add DateTimeNow() to the Download results separate macro output. Screenshot #2 shows the results with "Wait a Second" tool = 30 seconds. Rather than the timestamps being 30 seconds apart, they're all virtually identical, suggesting that the calls themselves are not being throttled.

I thought maybe I could avoid the issue by having the composed API call write out to a file via the Run Command tool, and use a command line cURL request to get the actual data. I tried this with a dummy .bat file instead of a curl command but the Run Command tool still wrote all 4 files simultaneously rather than being throttled. This suggests the curl call would also not be throttled.

Any ideas on how I can make this work?

[cgoodman3 · Accepted solution]

Hey @Scott_Snowman

I don’t have a copy of Alteryx in front of me right now, so attempting to answer this from memory. Instead of putting the wait a second macro outside the batch macro, have you tried putting it inside? I think this way, each record (api request string) will be sent into the macro but it will force it to wait at least a second before processing the next record.

i certainly have this set up on a few API requests that are limited and it works as expected. I also have learnt to write out the JSON response from within the batch macro, instead of at the end of processing.  if you set it up this way you can then monitor via file explorer that it’s behaving as you’d expect.

Let me know how you get on, and if that’s still not working I’ll have another think.

[Scott_Snowman]

@cgoodman3 This did the trick: putting the Wait a Second macro into the batch macro. Requests are coming out with 20 second pauses, matching the parameter set in the Wait a Second macro.

Thanks for the suggestion! I would have thought slowing the input to the macro was sufficient, but it appears I'm mistaken about how/when the various tools take action during a run.