Running multiple files one at a time from a single location through a workflow

[rmt1g13]

Hi all,

First time posting here! I have a workflow that runs successfully if I use a single file as my data input. My end goal is to be able to point my workflow at a specific file location containing many files, which will then be picked up and run one at a time through my workflow. I am having a lot of trouble trying to find the best way to do this, so any help or suggestions would be greatly appreciated.

[Jonathan-Sherman]

Hi @rmt1g13 and welcome to the community!

So two quick questions

(1) Are all the files you want to bring in the same structure?

(2) Do you want to run each file through the workflow separately or bring them all together and run them all through the workflow at once?

Regards,

Jonathan

[rmt1g13]

Hi Jonathan, thanks for coming back so quickly! 

(1) Yes they're all the same structure and format (.txt tab delimited).

(2) I need each file in the location to run through the workflow separately.

[afv2688]

Hello @rmt1g13,

You could read the files with the directory tool. If they all share the same format you can then input them wiht a dynamic input with the put the filename selected:

This will help you keep track of the files origin for later to be able to output them using the name in different files/folders etc.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Regards

[rmt1g13]

Hi @afv2688,

Thanks for the reply. The issue I'm having with this is each file from the location needs to go through the whole workflow (including the output) before moving onto the next file. The solution you've posted works great, aside from the fact it brings in all of the files together and then runs them as one output before splitting out each file right at the end.

I hope this makes sense! 

[Jonathan-Sherman]

Hi @rmt1g13,

You'll need to include a batch macro which feeds each filepath in one run at a time into an input tool. Therefore using just data from one file per run.

I'll mock up a sample.

If this solves your issue please mark the answer as correct, if not let me know!

Regards,

Jonathan

[rmt1g13]

Hi Jonathan,

This sounds great, thanks. Am I right in saying that this means the full workflow, including the output, will be complete for each file in the location one by one? 

I have limited experience with batch macros, so a mock up sample would help a lot thank you.

[afv2688 · Accepted solution]

Hello @rmt1g13,

What you need then is a macro to do the process in batch. The macro image is in the black box. This gets every file and process it one by one. This one does although the same calculations for each file.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Regards

[rmt1g13]

Hi afv2688,

This is it, thank you so much! 

A separate point, the workflow now in my batch macro will run for a single file in about 2 seconds, however, when I've now packaged it into the batch macro the run of that macro is projected to take longer than an hour. Is this normal?

[afv2688]

Hello @rmt1g13,

Glad it helped you. I am sorry but I am not sure if I am understanding you. Are you telling me, it takes you about 2 seconds to run the macro (doing a test run within the worfkow of the macro I suppose), but if you use the macro and do input the parameters it takes an hour? Just for the same single file?

If thats the case it is not normal. There must be something that you are missing somewhere. Processing the information in a batch macro takes longer than doing it in a workflow yes, but we are talking about a 10 to 20% longer (at max), not from 2 seconds to an hour.

Regards