Ingest multiple .csv files with varying fields

[craigedelman]

I'm trying to figure out a way to ingest many .csv files.  The situation I have is an Alteryx workflow that outputs daily a .csv with ~300 fields and 40k rows.... one per day.

Occasionally I have added and removed fields from this output over the past 300 days.

There is one field, that has not changed it's name or been removed over the course of the 300 days.  I need to ingest this field only.  Lets call this field.... "Current Commit Date".

I'm trying to ingest this "Current Commit Date" to see what a materials commit date was when it showed up on the .csv and compare it to what it currently is.

Example:

Inventory commits - 2022-01-01.csv

Inventory commits - 2022-01-02.csv

Inventory commits - 2022-01-03.csv

etc...

When ingesting this field, what I'm ultimately after is to get to a table like this:

'Date','Material','Current Commit Date'

'1/1/2022','123-456-789','3/1/2022'

'1/2/2022','123-456-789','3/5/2022'

'1/3/2022','123-456-789','3/5/2022'

Is there a way to do this?

Is there a way to only ingest one field?  If not, could it be possible if they were .xls files instead?

Thanks in advance!!!

[alexnajm]

I would use a batch macro and read in everything! This should help guide how to build it out: The Ultimate Input Data Flowchart - Alteryx Community entrée de données eingabedaten

[craigedelman]

Thanks for the tip.  I had actually discovered that link in my searching for a solution.  I tried working through option 3 but couldn't get it to work.  I'm assuming it's because I'm too rookie to figure it out.  Is there an even more-beginner tutorial of how to use those macro tools?

[craigedelman]

Actually let me clarify, i got it to work, but it only ingested files that had the exact same # of fields. (See screenshot)

[alexnajm]

Go ahead and use this macro in your workflow instead of running the macro by itself - a Directory tool can list all of the files and feed the FullPath into the Batch Macro.

Additionally the most important part is to make sure to do the last part of the article!

[craigedelman]

I think I set up up correctly.  Is there an additional setting I missed?

 

[alexnajm · Accepted solution]

You need to use this macro in a workflow! Save. the macro, open up a new workflow, put down a Directory tool, connect into the folder with the csv files, and then insert the macro into the workflow by right clicking on the canvas, going to insert, and going down to macro.

It's the set up in the Input Flowchart article - the only difference is you won't need the Formula tool in between since you are doing csv files

[craigedelman]

Ah yes!  thank you thank you thank you!  this was the missing piece.  Thanks for helping a newb.