Download Tool "Skip if there's a Website Error" Option

[Matt_Curtis]

Hi, I'm currently feeding about 4,000 URLs into the Download tool.  Some of these may be old websites that are no longer hosted.  As the records get passed, if the process runs into one of these (random, 1 in a 100 chance), the whole thing stops executing and errors out.

It would be fantastic if I could just tell it to skip any records that error out, instead of having the whole thing error out because of one dud website.  I don't know which ones are duds until I pass them through the tool, so there's no way to filter them out upstream.  Real Catch 22 here!

I have a coworker who's going to tackle this in Python instead, and he says he can put in a "try, except block" into his code.  His words, not mine, and I know nothing about python code...but given what I know about Alteryx, there must be a way to build this into the tool!

Thanks!

[SeanAdams]

This would be SUPER useful - the download tool really does need either an "Ignore all errors" or a retry option or both.

[rdoptis]

I too would very much like the ability to suppress errors (or convert to Warnings) from the download tool. The option to retry n times would also be helpful due to the some APIs having intermittent timeouts or brief periods of instability. 

Best regards,

Ryan

[apichart]

Do we have an alternative solution?

@Matt_Curtis  @rdoptis how did you overcome the issue?

[rdoptis]

@apichart- We ultimately implemented retry logic outside Alteryx via the Gallery API. Definitely not ideal or accessible to all Alteryx users. 

[Matt_Curtis]

@apichart  We recently overcame the issue using CReW macros, a Block Until Done tool, a standard macro, and a custom batch macro.  If you're familiar with the CReW macros, there is one called Conditional Runner.  Batch macros allow you to process data one record at a time, and the Block Until Done allows you to make sure things run in sequence; using all three, you can circumvent the issue.

There's a control workflow, a standard macro, and the custom batch macro.  In the control workflow, I've got a SQL query which pulls all websites I want to scrape.  I feed that list into my batch macro. Inside my batch macro is the Conditional Runner, which runs my standard macro.  I use the Block Until Done to orchestrate around the errors.  This is what the batch macro looks like:

So, each record (web site) comes through the control parameter one at a time and updates that text input.  The Block Until Done first writes that URL to a csv file locally.  Then, the Conditional Runner tool runs my Standard Macro, which is actually my original Download workflow that was hitting errors when I tried to do this 4,000 at a time.  Instead of trying to read every single one, I've now configured the input to read from the CSV we just wrote in BUD's step 1:

Then it writes out a new CSV that has the URL with some Boolean flags to see if that website has what I'm looking for.  Look back at the Batch macro and you'll see the '3WriteOut' file being pulled in and joined to a Successful run from the conditional runner.  A failure won't get joined and won't produce any output, it essentially just gets skipped.

Let me know if that makes sense or if you have any questions.

(BTW, I still think the Download tool should have some of this error checking process built-in even though I figured out how to do it this way.)