Alteryx Designer Desktop Discussions

Find answers, ask questions, and share expertise about Alteryx Designer Desktop and Intelligence Suite.
SOLVED

Batch Macro - process hundreds of large (350mb) files and write a new file for each

TS_Dan
5 - Atom

Screen Shot 2019-08-22 at 9.55.40 AM.pngI am struggling with creating a batch macro.  I see several posts on the subject, but I’m missing some basic understanding (I’ve never used a Macro in Alteryx before).

 

I have about 400 files in a directory, they are csv format, compressed. Each file is around 300 to 400MB, so wildcard isn’t going to work given it’s over 65GB of data.  File naming convention is 000000_0.gz thru 000395_0.gz.  All files have the same layout.  My task is to extract 13 of the 137 columns then write the results into separate files such as “new_000000_0.csv”  

 

My confusion is threefold:

  1. Do I need to build some kind of logic to identify each file by name? i.e. start with 00000_0.gz then add 1 to get 00001_0.gz and so on?  I am hopeful it will just automatically handle each file in the directory in sequence.
  2. In my calling workflow, what kind of input is “Input4” expecting?
  3. How exactly do I tell it to either write a single file for each input file? I think it’s somewhere in Interface Designer, but I can’t find it.

 

Here is my macro:

Screen Shot 2019-08-22 at 9.55.40 AM.png

And here is my calling workflow:

Screen Shot 2019-08-22 at 9.56.44 AM.png

1 REPLY 1
TS_Dan
5 - Atom
Labels