Download .gz file from Amazon S3 bucket

[ElisaAna]

Hi there, 

I'm trying to download some .gz files from an amazon S3 bucket. 

The S3 downloader tool from Alteryx is working, but .gz is not an option.

I've tried to use Aimpoint Digital's S3 toolkit, but I'm getting the following error message:  Error: Amazon S3 List (2): Connection was closed before we received a valid response from endpoint URL: xxxxxxxxx - the tool does not seem to need a specific an endpoint. Is that where I'm going wrong? 

I can't open the macros as I am on the wrong version of Alteryx (22.3 - and delayed in the upgrade). 

Thanks for any help

Elisa

[apathetichell]

How are you connecting to your S3 (ie key/value) or do you have an AWS role (ie have you pre authenticated on your system via cli)?

[ElisaAna]

Secret and access keys. I read a great article on the community which involved using command line tools to change directory to the AWS CLI location but I do not have any such location on my network. 

[apathetichell]

@alexnajm - any input on the aimpoint tool issue? I assume your python code doesn't specify file type and could be used for .gz files.

[alexnajm]

I assume so too - Tagging in @PhilipMannering / @BenMoss who will have a better answer than I do!

[ElisaAna]

Thank you! I can't get the aimpoint tool to work but I'm not sure it's querying the right endpoint. It has used the bucket name and turned it into : https://BucketName.s3.amazonaws.com/?encoding-type=url

And the error I get is:

Error: Amazon S3 List (2): Connection was closed before we received a valid response from endpoint URL: "https://BucketName.s3.amazonaws.com/?encoding-type=url".

Could it be a whitelisting issue? I'm in a very secure AVD.

[ElisaAna]

To clarify - I can see and download data from the bucket - but it can't read in the .gz files - it comes out as gibberish. The unzipped files come out fine.

[BenMoss]

Hi @ElisaAna sorry your latest comment is a bit confusing and suggests you have actually been able to download file files from the bucket, is that the case?

Ben

[ElisaAna]

Hi there, sorry I appreciate that it's confusing. 

I have used the Alteryx s3 connector and I can pull out unzipped .csv files fine. 

The files I do need are .gz files.

I've tried to use AimPoint Digital's s3 tools but they do not work - I think there is an error around the endpoint it is trying to query.

Thanks for your patience! 

[ElisaAna]

To update !

I had to whitelist the generic endpoint, then whitelist the endpoint that incorporates s3.eu-west-2.amazon.com and it's going through !! I can list files out fine.

I'm currently troubleshooting as there is a WinError 123 when I try and download a file. This error does not occur if the file is not in a path, but directly in the bucket.: 

i.e. filename.csv.gz works, but folder/folder/folder/filename.csv.gz

There are many special characters (. - : =) in the folder path and filename. Could that be it? How does the Aimpoint Digital tool handle special characters if I send the filepath in? 

Any clues?