Community Spring Cleaning week is here! Join your fellow Maveryx in digging through your old posts and marking comments on them as solved. Learn more here!

Alteryx Designer Desktop Discussions

Find answers, ask questions, and share expertise about Alteryx Designer Desktop and Intelligence Suite.
SOLVED

Input and process dbf files with different schemas

TheLord
5 - Atom

Hi everybody,

 

I have been struggling with the following task: I need to open about 1200 large DBF files (one has around 1.5GB), filter some records out and save each one into csv. I do this to reduce file size and later those csvs will be loaded into an sql database.

 

I planned to use a simple wildcard input, but given that some files have slightly different schemas, there is the " File XX has a different schema than the 1st file in the set and will be skipped" and "File XX has a different field name..." I used the macro described here https://community.alteryx.com/t5/Alteryx-Knowledge-Base/CS-Macro-Dev-Reading-in-Multiple-Files-with-..., but with no luck.

 

Since I don't want to join the files in this step, it is frustrating that Alteryx won't just filter and convert the files ignoring different schemas, as it doesn't really matter.

Capture.PNG

6 REPLIES 6
SamDesk
11 - Bolide

Hello @TheLord,

 

If you open your macro and go to View > Interface Designer and then click on the cog, have you tried setting one of the Auto Config options and/or selecting the tick box that the output will change?

 

Capture.PNG

 

Sam :)

TheLord
5 - Atom

Hi @SamDesk,

 

thanks for replying. I have actually tried to switch the "input file" for "directory" in the main workflow (the one above), and it seems to work. However, I have another issue - the macro is running, but I cannot say if it works as I want it to. I should get 1200 csvs as a results, but the macro does not output one by one (screenshot of the macro below). Can I somehow force it to output one file at a time, or at least see which file it's processing now?

Capture.PNG

 

SamDesk
11 - Bolide

Hi @TheLord,

 

Unless you set up the output in your main workflow to dynamically change the output it will output everything to the same file. What you can do is use a formula tool to create a full path for each group of records based on common identifiable fields between those records. And then in the output tool use the very bottom option to change the full file path based on a field. I've created a simple example of this in the attached and screenshot below.

 

Capture1.PNG

 

Also inside your macro, you need to replace the output tool with a macro output tool so it can pass to the output tool in your main workflow.

 

Capture.PNG

 

Sam :)

TheLord
5 - Atom

Hi @SamDesk,

 

thanks, that looks good. However, that solves a problem of writing the output into 1200 different files. But Alteryx will do so after the whole macro is finished - since the data have around 1TB, I would like to get the csv files one by one, not after the whole process. If something goes wrong, I will lose all the data.

 

Michal

SamDesk
11 - Bolide

Hi @TheLord,

 

In that case, you would want to put the normal output tool inside the macro, using the new dynamic file path. It then should output one by one.

 

Sam :)

TheLord
5 - Atom

Hi @SamDesk,

 

thank you very much, that works! The files output to my desktop, not the specified folder for some reason, but that is a detail I will fix easily I hope :)

 

Michal

Labels