Input and process dbf files with different schemas

Hi everybody,

I have been struggling with the following task: I need to open about 1200 large DBF files (one has around 1.5GB), filter some records out and save each one into csv. I do this to reduce file size and later those csvs will be loaded into an sql database.

I planned to use a simple wildcard input, but given that some files have slightly different schemas, there is the " File XX has a different schema than the 1st file in the set and will be skipped" and "File XX has a different field name..." I used the macro described here https://community.alteryx.com/t5/Alteryx-Knowledge-Base/CS-Macro-Dev-Reading-in-Multiple-Files-with-Different-Field/ta-p/13370/label-name/batch%20macro/page/2/show-comments/true, but with no luck.

Since I don't want to join the files in this step, it is frustrating that Alteryx won't just filter and convert the files ignoring different schemas, as it doesn't really matter.

Batch Macro

Accepted answers

SamDesk

Hi @TheLord,

In that case, you would want to put the normal output tool inside the macro, using the new dynamic file path. It then should output one by one.

Sam

All comments

SamDesk

Hello @TheLord,

If you open your macro and go to View > Interface Designer and then click on the cog, have you tried setting one of the Auto Config options and/or selecting the tick box that the output will change?

Sam

TheLord

Hi @SamDesk,

thanks for replying. I have actually tried to switch the "input file" for "directory" in the main workflow (the one above), and it seems to work. However, I have another issue - the macro is running, but I cannot say if it works as I want it to. I should get 1200 csvs as a results, but the macro does not output one by one (screenshot of the macro below). Can I somehow force it to output one file at a time, or at least see which file it's processing now?

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703