Tool Mastery | Amazon S3 Upload and Download

[EddieW]

This article is part of the Tool Mastery Series, a compilation of Knowledge Base contributions to introduce diverse working examples forDesigner Tools. Here we’ll delve into uses of the Amazon S3 Tools on our way to mastering the Alteryx Designer:

The Amazon S3 Upload Tool and Amazon S3 Download Tool and are connectors that allow you to upload and download files to/from your Amazon S3 account with Alteryx Designer. You do not need to install an ODBC driver to use these tools, and they come with the default installation of Alteryx Designer or Server. You can find them conveniently placed under the Connectors tool category. This Tool Mastery Article is here to help you learn how to connect to your Amazon S3 account with Alteryx like a pro!

To upload data to Amazon S3 using Amazon S3 Upload Tool:

1. First, you need to make sure you already have an AWS account and credentials.
2. If you have step one covered, simply drag the Amazon S3 Upload Tool onto your designer canvas. The configuration window for the tool will load on the left-hand side of the canvas.
3. To configure the tool, fill out the options in the configuration window as detailed below:

1 -  an AWS Access Key is required to connect to your S3 account with these tools. You will need to reach out to your AWS admin if you don't know what your access key is.
2 -  an AWS Secret Key is also required to use this tool. Check with your AWS admin if you don't have the secret key.
3 - Expanding the drop down will give you 3 options for you to select from:

a. Hide (Default) - will hide the provided password using minimal encryption.
b. Encrypt for Machine - Allows any user on the computer to fully use the connection.
c. Encrypt for User - Allows the logged in user can use the connection on any computer.

4 - Select Default from this drop-down menu to allow Amazon to determine the endpoint automatically based on the bucket you select. To specify an endpoint for private S3 deployments, or if you know a specific bucket region, you can alternatively select an endpoint (S3 region), enter a custom endpoint, or select from one of ten previously-entered custom endpoints.
5 - You can select this option to use Signature Version 4 instead of the default Signature Version 2. Selecting Signature Version 4 will increase the connection's security but may cause the connection to be slower.
6 - Type a Bucket name to upload your data to a new bucket, or select one from the list of available Buckets.
7 - Specify the Object name (data file) to be stored in the above-specified Bucket.  You can either type an Object name or select one from the list.
8 - Server-Side Encryption is a drop-down option where you can select an encryption method for uploading to an encrypted Amazon S3 bucket
9 - The final configuration option allows to specify the File type you want to upload your data to S3 as.

4. Run the workflow. If the workflow runs successfully, you will see the message below in the results window.

To download data from Amazon S3 using Amazon S3 Download Tool:

1. Drag the Amazon S3 Download Tool to the designer canvas.
2. You will notice the configuration is identical to the configuration of the Amazon S3 Upload Tool (detailed above). If you need help with the configuration, please refer to the screenshot above or the Amazon S3 Download Tool Help page.
3. After you have configured the tool to access your intended file, run the workflow.
4. When the workflow is finished running, you will notice data from the Object has been imported into Designer 

There is also a great video that provides an overview of this information available here.

Some of the common errors (or special request) you will run into with this tool:

Issue: S3 – Error from AWS: The HTTP version specified is not supported

Answer: Make sure any files you are downloading, or uploading do not contain whitespaces or colons.

Issue: Cannot read in more than one object at a time

Answer: The Community Knowledge Base Article Amazon S3 Download – Use Wildcards to Select a Single or Multiple Objects goes over how you can utilize a batch macro to read in the list of objects in a bucket and allow you to filter for the file(s) you want using wildcards.

Issue: To utilize Amazon S3 Tool to access a subfolder within a private External Bucket in S3.

Answer: Refer to this community thread.

If you'd like some additional tips and tricks on connecting to Amazon S3, please take a look at the following resources:

Download a Zipped Excel File from an Amazon S3 Bucket

Extending Integration with AWS

By now, you should have expert-level proficiency with the Amazon S3 Download/Upload tools! If you can think of a use case we left out, feel free to use the comments section below! Consider yourself a Tool Master already? Let us know at community@alteryx.com if you’d like your creative tool uses to be featured in the Tool Mastery Series.

Stay tuned with our latest posts every Tool Tuesday by following Alteryx on Twitter! If you want to master all the Designer tools, consider subscribing for email notifications.

[sathiya]

Hi Any plans to add Short term keys to the tool - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html , for security reasons long term keys are not allowed. Thanks

[EddieW]

 Hi @sathiya,

Thank you for the reply! I am not aware if our development team has any plan to add short term keys to the tool. I would recommend you to add this feature request to our Idea page so our development team aware this is a feature our customer is looking for and potentially implement in future releases. 

Regards, 

[Sharmila]

Hi 

Can somebody help me upload one excel file with two sheets ( different schemas) to an S3 bucket using S3 upload?

[slacson]

Hello, I can't see the Amazon S3 tools in my Connect tab.  The only option I have is ODBC loader.  Adding additional tools doesn't have any options for me to add to the Connect tab.  Has anyone had this issue?

[kirkpoucher]

Is there any way to use an S3 file as an external source for a Drop Down tool?

[jmanduca]

The link to "AWS S3 access from EC2 Instance using IAM roles" goes to a "access denied" page - is the link broken? This is something I want our users to use over access keys, which is best practice when running within AWs.

[NeilR]

@jmanduca that article has been archived, but here's what it said:

Using an IAM Role to Grant Permissions to Alteryx Designer or Server, Running on Amazon EC2 Instances:

1. Create an IAM role in AWS with required access to AWS services
2. Attach this role to the EC2 instance (Alteryx AMI) at launch time or while the instance is running
3. Install the AWS CLI on the EC2 instance if necessary
4. Open a command prompt
5. Run the aws cli command to test access via the IAM role
6. Open Alteryx Designer
7. Go to Developer Tools
8. Drag the tool Run Command to the canvas
9. Configure the tool to run the aws cli command
10.  Run the workflow

[dhofberg]

in the upload to S3 bucket tool, what happens when the workflow is run a second time, does the object just get overwritten, or does S3 save it as a separate object even though the name hasn't changed. 

I don't see an option in the output tool configuration?

[alt19790525]

@dhofberg - Yes, it will overwrite the file/object name in S3 bucket when you run the workflow again.

[aehrenwo]

Is there anyway to get the last write date or updated date of the file using the S3 Download connector?  We were previously using files from a Shared Drive and getting that meta data was pretty straight forward. Not sure how to replicate with the S3 connector. 

[Lazzanova]

"Specify the Object name (data file) to be stored in the above-specified Bucket.  You can either type an Object name or select one from the list."

Unfortunately, if I try to select one from the list, Alteryx crashes... 

Everything else works perfectly fine. I can select a Bucket, so the Access Key and Secret Keys are set right....

Last week I was able to select a JSON file, but now I can't. Nothing seemed to be changed in the S3 Bucket. 
I did however reinstall Alteryx once... But that shouldn't cause this to happen right?

[Goppi]

Why did you constraint the file types to the 4 and not allow any file type? In my case, I would like to use it to upload images and sql files on a regular basis, but in general, S3 can be used with any file types. Is there a way around those restrictions?

Thanks,

Goppi

[sarojini]

I would need to use Amazon S3 download tool to bring in avro file but it shows error in file format might is that the reason being the file serialized/binary format file? how to handle such scenarios ..

Please share the steps to solve this problem else I have no option other than running into performance issues by using Python tool since avro file is huge.. 

[Goppi]

@sarojiniI have not found any solution for my problem using the Alteryx tool. Instead - which turned out to be also quite a bit faster for different reasons - I created in AWS an API gateway to S3 and use the standard Download tool in Alteryx. This allows me to perform any actions on any file type. The only caviat is that the API Gateway has a size limit of 10MB for the payload - which doesn't affect me, though i assume this will render this solution as useless for you.That said, the reason I am using the API gateway is amongst others because i needed to have control on a bucket level and source access. If that is not a requirement, you could use the S3's RestAPI which supports in simple form files up to 5GB and by using the multipart upload API, objects of up to 5TB.

There are tutorials available in the Internet on how to configure an API gateway - and of course, the API gateway is not for free. I don't have the link to the one i used anymore - but if you follow this path, ensure you use a tutorial that is already absed on the virtual hosted style URLs. The one i used was based on the path-style URL and AWS wants to deprecatiate this style since last year.

Hope that helps.

[Sharmila]

Hi,

I have workflow that generates a.csv file with a specific naming convection to upload to a S3 bucket that will be used by a lamda function in AWS to another process.

Anybody can assist me on how to upload the .csv file generated with that specific name uploaded to S3 bucket immediately in the same workflow?

Thanks for your help in advance.

[aahaseebkhan]

How fast does it read from S3? is it as fast as reading Input data tool?

[akasubi]

Hi, is it possible to upload .zip files to S3 via this tool?

[afletemi]

How do I know if data being loaded to S3 bucket is encrypted in transit?   Is there anything I can provide IT to prove the tool is using HTTPS and not HTTP?