AWS S3 upload error

A little unsure - maybe it's a network issue?

Assuming this is the regular Amazon S3 Upload Tool (alteryx.com), have you tried the AWS tools offered by Aimpoint Digital on the Marketplace? AWS Tools by Aimpoint Digital | Alteryx Marketplace

It has completed when our network has been slower, so I don't think that's the issue. It is the normal S3 upload tool, I have not looked at the other tools before.

what's the file size? can you auth into aws cli locally (and if so - can you upload with an aws s3 cp {{filename}} {s3 bucket} command?

file size is 8 gb. we can upload on the machine and locally through the tool no issues, it doesn't fail all the time, just occasionally on bigger files

This is probably a network timeout issue then. because of the file size. Any chance you can partition the file and upload in bunches?

I looked into the new S3 tools, and can't get the configuration to work. I've filled in the AWS access key id and secret access key, and the file paths, but I'm getting an error: "No module named 'boto3'. I've attached a picture of the base S3 tool since I don't want to post a picture of the keys, but I have filled in all the blank spaces and it isn't uploading. Do you know what configuration element I'm missing?

s3tool.png

Run as admin. If that doesn't work. open a new workflow as admin. add a text input too. add a python tool. in your python tool add the following:

from ayx import Package
Package.installPackages(['boto3'])

run it. re-run your s3 tool.

you need to download the boto3 python package. @alexnajm - I'd recommend a different implementation here.

@apathetichell I trust your opinion, just offering up alternative solutions in case it alleviated the issue! I agree batches may be the way to go

Tagging in @PhilipMannering who I think worked on building these Marketplace tools in case there's anything else I may be missing

Uploading in batches isn't an option for me, but thank you for the options and help. I will try this solution, and look forward to the help on the S3 tool.

@JohnLight - ok - just make sure you set up boto3 in your alteryx python venv in order to use the Aimpoint version

I asked a few people, and they agree that the 8GB file size is probably the issue, so unsure if our tool will solve it. Why isn't batching an option?

Either way make sure boto3 is installed to try out the other version!

Boto3 is installed, and I have it working with a test text input tool. How would I go about doing it in batches? Unioning 5+ csvs in quicksight (which is where we're using the data after uploading it to S3) defeats the purpose of convenience of using the S3 upload tool. I've also been able to upload files that are much larger up to 10x this size on similar network speeds.

o.k. - one more question - any chance you can run the workflow on an ec2 versus locally? that would prevent any kind of network timeout (assuming the aws role assigned to the ec2 has direct access to the s3). This would require LICENSE SERVER. Not a SERVER LICENSE - LICESE SERVER.

with these file sizes - I'd probably batch them and set up a Lambda job to union files triggered when a new file was added the s3. I haven't used Quicksight - so not sure if you can use a Glue metadata store to build an artificial table off of the individual files - which would be other avenue.

What is the benefit of the new tool over the old tool?

The main benefit is that it has the Amazon S3 List tool to grab a list of objects from your bucket. Not sure if that's part of your process at all. But the main reason I suggested the alternate tools was because maybe they would work without needing to batch.

If you are still finding issues, I would follow @apathetichell 's lead since there seems to be more direct experience with S3 than i have.