Multi-Threaded Download

[mdata]

Alteryz 10.0 it says it supports multi-threading for the download tool. How do you enable it? I see the setup right now it’s making sequential HTTP request instead of multi-threaded mode.

[jdunkerley79]

Go to the Connection settings of the download tool and you can set the Maximum Connections to download multiple threads at once.

[mdata]

Thanks! Can it you do it for a single file? For example, I’m taking record (row) from a file of 10000 records and making http request using the download tool. Is it possible for it take 32 records at time instead having 32 inputs?

[mdata]

Thanks! Can it you do it for a single file? For example, I’m taking record (row) from a file of 10000 records and making http request using the download tool. Is it possible for it take 32 records at time instead having 32 inputs?

[mdata]

Thanks! Can it you do it for a single file? For example, I’m taking record (row) from a file of 10000 records and making http request using the download tool. Is it possible for it take 32 records at time instead having 32 inputs?

[jdunkerley79]

COuld you read the entire file into Alteryx in one go and them process?

If you can post your workflow happy to take a look

[mdata]

Attached are the workflow and the macros. I would like to create a loop when reading from a file and process x number of records and append it to a file and repeat the process until all records are done. For example, I have 46 records in this file then it takes the record and creates a json array and uses the download tool and writes to a file. I would like to process 5 records from the files at time instead all 46 records them process them and append to the results file. Ideally I would like this to be muli-threaded, if not I can process it as a batch records of 5 at a time.  Thanks!

[mdata]

I can use the select tool to select the first n records. I would like to repeat it until all the records are processed and append it to a single file at the end.

[jdunkerley79 · Accepted solution]

Having taken a quick look, you can use an iterative macro to batch it up.

I threw together but havent tested an iterative macro as a start for you

Samples first 10 and passes to download, then loops while still content. Msacro will batch together the outputs.

Hope it helps

[mdata]

Thank you.