Faster download connector -- throughput of download connector is too low.

[gbtimmon]

Hello,

Im looking for a little guidance -- We have an internal service where I work that is high-latency high throughput. That is every request takes about 200ms to return an answer, but the service can happily handle thousands of request at the same time, reutrning all in about 200ms. If I had a thousand rows of data to process I can send them all at roughly the same time and process them in about 200ms. 

Heres the rub -- the download tool only lets me send 32 max concurrent connections. This mean that if I have a 1000 rows of data my processing needs roughly (1000/32)*200. With java messaging queue we regularly put 100k rows through this service in under 30 minutes. With the alteryx download tool I need about 15 mins to run 4k rows. 

Is there any way I can speed up the download and increase the number of concurrent connections? Ive found multiple download tools will not run concurrently even if I split the data.  

Im looking into writing a faster service connector in R, but I wanted to avoid that if possible. 

Any thoughts ??

[NeilR]

With help from Stackoverflow I created a workflow (attached) that creates and executes a batch script that in turn kicks off concurrent curl processes, then waits for each to finish before exiting. Alteryx then reads the downloaded data back in and presents it as if it came out of the Download tool. Hopefully you can edit it to suit your needs.

Alternatively, you can do some hacking to squeeze out a few more threads from the Download tool. In the Advanced tab of the User Settings, check "Display XML in Properties Window".

Then in your Download tool configuration, click on the XML View.

Click Edit in the top right corner, and edit the numConnections tag. I was able to get it up to 63 before getting an error.

[tmlmark]

@NeilR Wow, that was amazing. I was able to up the connection count on mine to 32 and this increased the speed by several factors. 

[NeilR]

@tmlmark good to hear. Keep in mind you can set the connection count to 32 in the UI (see image below). My "hack" above was to get it higher than 32.

[m_v]

It doesn't look like the settings are saved after I modify them in the download tool. It always reverts back to 32.