Perform workflow on all files in a folder separately and output a new file each

Dear community,

you have been beyond helpful already but I would like to ask you for support one more time.

So I have a folder with a lot of big sized csv files. I created a workflow which I want to use on each of the files to clean them. The goal is to have a new cleaned file (with the same file name) in another folder for every file in the input folder. I attached my workflow to give you an understanding.

If I use the batch macros I found here in the forum, they will just union all files together, which will produce wrong results with my workflow. I think unioning everything and splitting it up again is not an option. I am looking for a way to truly run the workflow separately on the files without having to manually input the new file everytime. Also if I try to use the batch macro on my workflow I get the message: Error: Macro for multiple CSV files (2): Record #2: Tool #7: The field schema for the output "Output7" changed between iterations.

Maybe I just didn't get the right combination of Directory, Dynamic Input and Batch Marco Tools for my purpose.

Is someone able to give me some advice? Thank you so much!

hasi82

Workflow upload.yxmd

Batch Macro

Help

Accepted answers

Thableaus

@hasi82

Here's an example:

The Batch Macro is equivalent to your workflow. You need to add, as I said before, the Control Parameter, a Formula Tool and a Folder Browse Interface Tool.

Change the Output Tool to Take File/Table Name as your Field FullPath recently created and make it change the Entire FilePath

Go to the workflow, take the Directory tool and specify your folder.

In the Batch Macro you created, just update the Folder you want to save each file.

Let me know if you have questions. Flow appended with Macro (need to change inputs/outputs).

Cheers,

MacroBatchExample.yxzp

All comments

Thableaus

Hi @hasi82

There's no need to union records.

The thing is you should transform your workflow into a batch macro.

Then, use Directory Tool to locate the folder where your csv files are in a different workflow to update control parameter in your Input Tool.

You could change the Output Folder using Formula Tool + the option to change file path with Take File/Take Table Name field.

It's a very simple task, I just don't have any example to do it for you now, but if you learned about batch macros, you should be able to do it with no problems.

Cheers,

Thableaus

@hasi82

Here's an example:

The Batch Macro is equivalent to your workflow. You need to add, as I said before, the Control Parameter, a Formula Tool and a Folder Browse Interface Tool.

Change the Output Tool to Take File/Table Name as your Field FullPath recently created and make it change the Entire FilePath

Go to the workflow, take the Directory tool and specify your folder.

In the Batch Macro you created, just update the Folder you want to save each file.

Let me know if you have questions. Flow appended with Macro (need to change inputs/outputs).

Cheers,

MacroBatchExample.yxzp

hasi82

@Thableaus thanks a lot! With your help I managed to get it running

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703