Extract data from csv file inside a gz file from Download Tool

Hi guys,

I need to read and process data from a csv file that is contained inside of a GZIP file that I need to download daily from the url: https://github.com/marcuswac/covid-br-data/raw/master/covid-br-ms-complete.csv.gz

I don't need to save the zip file, I don't need to save the csv file. I just need to extract the data inside of that file and use it within a workflow.

Is there a simple way to incorporate the Download Tool with something to Extract+Parse data from this link into a flow??

I can not figure this out!

Please help!

Parse

Download

Accepted answers

Maskell_Rascal

Hey @pperrot1

Not sure how familiar you are with Python, but this is exactly the kind of thing I do with that tool.

First input the URL to the Download tool and configure it to download to a Temporary File.

I then use a Select tool to select only the path to the temp file and rename the field.

Its now ready to be connected to the Python tool, so the path will be read into the code I built.

Final output from Output #1 looks like this:

I've attached a workflow for you to try out.

If this solves your issue please mark the answer as correct, if not let me know!

Thanks!

Phil

Download and Extract GZIP File.yxmd

All comments

BrandonB

This workflow should work for you. I leveraged a batch macro after the download tool which has a dummy control parameter in it to force it to execute second. This process does download the file to your machine based on the path specified in the text input, however, this will be overwritten daily so it shouldn't take up a lot of space on your machine. Packaged workflow is attached and I have also included both the workflow and the macro in case you have issues extracting.

download 1.png

download 2.png

download and extract.yxmd

download and extract.yxzp

Input Macro.yxmc

Maskell_Rascal

Hey @pperrot1

Not sure how familiar you are with Python, but this is exactly the kind of thing I do with that tool.

First input the URL to the Download tool and configure it to download to a Temporary File.

I then use a Select tool to select only the path to the temp file and rename the field.

Its now ready to be connected to the Python tool, so the path will be read into the code I built.

Final output from Output #1 looks like this:

I've attached a workflow for you to try out.

If this solves your issue please mark the answer as correct, if not let me know!

Thanks!

Phil

Download and Extract GZIP File.yxmd

BrandonB

Awesome approach @Maskell_Rascal! Your solution is nice because it leverages the temp file rather than requiring one to be downloaded. I know that a regular Input Data tool can be used with a .gz file, but I haven't had luck with a Dynamic Input tool on a temp file. Could be something I'm missing in the configuration, but in any case, your solution looks perfect.

@Maskell_Rascal your Python solution seems to be what I need for a problem I'm facing. But is there a way to get the Python tool output to include the filename of the downloaded file? I'm not very familiar with Python.

Quick Links

This months top contributors

mceleavey 383

mbarone 337

Hollingsworth 335

LanisC 335

RithiS 280