How to open multiple zip files with Input Data Tool?

Does anyone know how to input multiple zip files with Input Data Tool?

I am trying to input 2,300 zip fies(all of them have one csv file in it and they are in the same folder) with Input Data Tool, but cannot find the way to select multiple zip files in "Open a data file" window.

I know that we can open mulple csv files using " * ", however it did not work for zip files.

Please let me know how to figure it out.

Input

Accepted answers

jdunkerley79

I think you can do this using a directory tool to get a list of all the zip files and then feed it into a dynamic input tool updating the filename as long as the csv files are named the same within the zip.

I have attached a sample showing how it might work. You will need to change the CSV filename in the formula tool if this would work

Otherwise if filenames vary, you could do this by extracting all the files into a temporary folder and importing the results.

I am happy to put a sample together for the second if that is the case

dynamic input zips.yxmd

All comments

jdunkerley79

I have attached a sample showing how it might work. You will need to change the CSV filename in the formula tool if this would work

Otherwise if filenames vary, you could do this by extracting all the files into a temporary folder and importing the results.

I am happy to put a sample together for the second if that is the case

dynamic input zips.yxmd

shradhaahir

I have 6 zip files(names differs) within which there are csv file with same name as zip file.

I want to collate all csv file into one file.

Wilcard path is not working. how to use dynamic input here?

JamesChiswell

Could you please put a sample together for the second case where the filenames vary.

Cheers

James

TeePee

Thanks! This was super helpful. Here's how I tweaked it for reading in multiple .csv.gz files (again, provided they all use the same schema). It only takes a couple of minutes. I love standing on the shoulders of giants 🙂

Here's a screenshot of jdunkerley79's flow:

Step One: Directory Tool

Amend the directory to point to the folder containing the .csv.gz files.
Amend the File Specification to say *.csv.gz

Step Two: Formula Tool

Amend the Formula tool text to say this exactly (don't make any other changes): [FullPath] + '|||Query=datafile.csv'

Step Three: Directory Tool

First, click on the Directory Tool's Edit... button:

Then point the "Connect to a file..." field to any one of the .csv.gz files in your folder.
Check the "2 File Format" says Gzip
Check the "3 File in Archive" says datafile.csv
If you'd like to check which files have been loaded (eg using a Summarize tool), make sure to select "File Name Only" as the option for "5 Output File Name as Field." This is useful for troubleshooting; if a file accidentally has a different schema you won't see any other errors or warnings. Optional but highly recommended.
Check "6 Parse Selected File as" says *.csv
Finally check over all the other selections to ensure they work with your particular data set (eg does your data have headers? does the data start on the first row? etc)

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703