Hi everybody,
I have been struggling with the following task: I need to open about 1200 large DBF files (one has around 1.5GB), filter some records out and save each one into csv. I do this to reduce file size and later those csvs will be loaded into an sql database.
I planned to use a simple wildcard input, but given that some files have slightly different schemas, there is the " File XX has a different schema than the 1st file in the set and will be skipped" and "File XX has a different field name..." I used the macro described here https://community.alteryx.com/t5/Alteryx-Knowledge-Base/CS-Macro-Dev-Reading-in-Multiple-Files-with-..., but with no luck.
Since I don't want to join the files in this step, it is frustrating that Alteryx won't just filter and convert the files ignoring different schemas, as it doesn't really matter.
Solved! Go to Solution.
Hello @TheLord,
If you open your macro and go to View > Interface Designer and then click on the cog, have you tried setting one of the Auto Config options and/or selecting the tick box that the output will change?
Sam :)
Hi @SamDesk,
thanks for replying. I have actually tried to switch the "input file" for "directory" in the main workflow (the one above), and it seems to work. However, I have another issue - the macro is running, but I cannot say if it works as I want it to. I should get 1200 csvs as a results, but the macro does not output one by one (screenshot of the macro below). Can I somehow force it to output one file at a time, or at least see which file it's processing now?
Hi @TheLord,
Unless you set up the output in your main workflow to dynamically change the output it will output everything to the same file. What you can do is use a formula tool to create a full path for each group of records based on common identifiable fields between those records. And then in the output tool use the very bottom option to change the full file path based on a field. I've created a simple example of this in the attached and screenshot below.
Also inside your macro, you need to replace the output tool with a macro output tool so it can pass to the output tool in your main workflow.
Sam :)
Hi @SamDesk,
thanks, that looks good. However, that solves a problem of writing the output into 1200 different files. But Alteryx will do so after the whole macro is finished - since the data have around 1TB, I would like to get the csv files one by one, not after the whole process. If something goes wrong, I will lose all the data.
Michal
Hi @TheLord,
In that case, you would want to put the normal output tool inside the macro, using the new dynamic file path. It then should output one by one.
Sam :)
Hi @SamDesk,
thank you very much, that works! The files output to my desktop, not the specified folder for some reason, but that is a detail I will fix easily I hope :)
Michal