Time Delay on record processing

[BetterFerret]

I will try to make this short but the back story is a bit long.

I was recently tasked with scraping a website requiring repeated call to the URL with about 10,000 different queries. Pushing all 10,000 at the Download tool caused intermittent DownloadData to be returned with HTML from what appeared to be a default fallback help page.  Not what I needed.  I suspected the site may have seen all the calls in rapid succession as a DDoS attack or something, so I put a Throttle tool in line to lessen the burden on their server.  It reduced the failed calls, but there was still more than I found acceptable, requiring pulling out the failed queries and repeating the same throttled processing.  Putting time between each record was what I needed.  And then I found this Wait/Pause Between Processing Records  Just what I needed.

Now the constructive criticism.  I hope I don't offend anybody.

The macro does the job using a simple ping statement inside a grouped batch macro, pinging until a selected time interval has passed.  It does this repeated pinging for every record.  That can add up to a lot of pings especially if the time interval is rather large and a lot of records are being processed.  Then DDoS popped into my head.  The same issue that led me to find this very solution.

So, I started thinking how could I accomplish this same wait between records and iterative macro seemed plausible, seeing that loops can be used in code to do this very thing.

I have attached the macro I came up with.  Feel free to check it out, critique the hell out of it, and/or used it if it will solve any problem you might have.  I only ask that you keep the macro intact and give credit where credit is deserved.

Thanks for your time,

Dan Kelly

Time_Delay.yxzp

[BenG]

Hi Dan,

Thanks for the post.  Just trying to understand the request on this.  I understand that the Wait/Pause between records can be implemented better, but how can the Throttle tool be improved to solve the use case you are describing above.  My assumption is that slowing down the requests enough should  provide enough time between records to avoid the failures from too many requests.

Thanks,
Ben

[BetterFerret]

Hey Ben,

The problem with the Throttle tool is that it seems to 'group' the records let through per minute.  If I set it to 60 (trying to get 1 per second) the result is 60 records output downstream per minute all at once, not one per second.  Now, if the Throttle had more granularity (per second instead of per minute) it would be doing what I needed, actually allowing one record to be processed down stream per second.

I hope this helps explain.

Dan

[KylieF]

Thank you for posting to the Alteryx Community! Your idea is interesting to us, but we have determined that we are unable to place this idea on our road map for the product due to several factors. However, should we be able to return to your idea in the future we will update the status back to Under Review.