Download tool creates an empty file after HTTP download header error code

[ColinG]

Hi,

So I'm using the Download tool to call an API that allows an authorized user to download Excel files from a server. I have the input to the Download tool sending it 6 reports to download and write the output to a filename in an input field.

Some of my files are downloading to my local file system with a 200/Success download header response in Alteryx. Those Excel files open and display their data fine.

Some of my files show up into the folder they are configured to save into, but Alteryx shows a 429/Failure download header. The files exist on my local file system as 1 KB files, and if I try to open them, Excel says they are corrupted.

429 is the error code for too many concurrent connections and it could vary based on current server load. I have not been able to figure out how to rate limit Alteryx across many tools/workflows since the Throttle tool doesn't really work. I think I can theoretically convert this to a macro and send each of the files into the macro individually to be downloaded and throttle or time delay it that way. But that is not really my question/curiousity.

I'm wondering when the Alteryx download tool does not get a 200/Success header, and it is set to output to a filename from a field, why does it store a file at all? Shouldn't it not write the file until it gets the 200/Success, or if it is raw writing the bitstream at least delete the file by default if the download header response isn't 200/Success?  Is there a way to configure it to do this within the download tool, without tracking the responses and then doing a toolchain to delete any files that were written without a 200/Success download header?

Thanks.

[apathetichell · Accepted solution]

You have the download tool configured to write a file. The download tool does not differentiate between the 200/binary response and the 429 1k message response. It writes the 429 message. add a filter tool/second download tool to retry the 429s.

[ColinG]

It is silly that I didn't think of trying to open the 1kb file in Notepad++ to see what the raw file contents is. Thanks for the hint. You are correct it is the header contents instead of the bitstream it didn't receive.

I still think it is inconsistent and not best practice that the behavior of the download tool changes when using the write file option - to writing a file of the download header results only when it doesn't get the 200/binary response, which is duplicated in the tool output. But, if it gets the 200/binary response it does not write the header to a file (only to the tool output) and instead writes the binary file. It would make more sense if it always wrote a header file seperate and in addition to any bitstream it actually did receive. I would have developed the tool to do either optionally but not mix data types into one file output depending on a result code.

The fact it mixes these concepts means the development of my "applet" got an order of magnitude more complicated, but it isn't the end of the world.

Your idea is a reasonable partial solution, but it would not always work as I add more files into the download list, unless it was also recursive and would do that over and over again until all desired files downloaded, even the 2nd try didn't get all the files. 

It looks like the service may allow up to 3 concurrent connections generally, but if server load is high, it can fail at 3 or 2 connections. I believe Alteryx is spawning download connections as quick as it can up to some (perhaps memory) limit, and how many connections are ongoing just depends on how quickly prior downloads completed. 

I've already added a step where I save all the file names to a table that I got a 200/Success header from, meaning I should have good files saved, and so if I rerun the flow it will only retry files that were not successful. To have a robust app that doesn't leave file system garbage I need to add a feature to delete files that are left that did not get a 200/Success, and I'll probably turn it into a macro so I can actually rate limit concurrent downloads to hopefully prevent this in most cases.

[apathetichell]

I use an iterative macro with a batch macro controlling concurrency (so yes -> recursive in Alteryx terms) for these kinds of situations. Max concurrency is 32 on the download tool.