Advent of Code is back! Unwrap daily challenges to sharpen your Alteryx skills and earn badges along the way! Learn more now.

Alteryx Designer Desktop Discussions

Find answers, ask questions, and share expertise about Alteryx Designer Desktop and Intelligence Suite.
SOLVED

Batch Macro - process hundreds of large (350mb) files and write a new file for each

TS_Dan
5 - Atom

Screen Shot 2019-08-22 at 9.55.40 AM.pngI am struggling with creating a batch macro.  I see several posts on the subject, but I’m missing some basic understanding (I’ve never used a Macro in Alteryx before).

 

I have about 400 files in a directory, they are csv format, compressed. Each file is around 300 to 400MB, so wildcard isn’t going to work given it’s over 65GB of data.  File naming convention is 000000_0.gz thru 000395_0.gz.  All files have the same layout.  My task is to extract 13 of the 137 columns then write the results into separate files such as “new_000000_0.csv”  

 

My confusion is threefold:

  1. Do I need to build some kind of logic to identify each file by name? i.e. start with 00000_0.gz then add 1 to get 00001_0.gz and so on?  I am hopeful it will just automatically handle each file in the directory in sequence.
  2. In my calling workflow, what kind of input is “Input4” expecting?
  3. How exactly do I tell it to either write a single file for each input file? I think it’s somewhere in Interface Designer, but I can’t find it.

 

Here is my macro:

Screen Shot 2019-08-22 at 9.55.40 AM.png

And here is my calling workflow:

Screen Shot 2019-08-22 at 9.56.44 AM.png

1 REPLY 1
TS_Dan
5 - Atom
Labels