s3 AWS Redshift upload connector

[adrianloong]

Given redshift prefers accepting many small files for bulk loading into redshift, it would be good to be able to have a max record limit within the s3 upload tool (similar to functionality for s3 download)

The other functionality that is useful for the s3 upload tool is ability to append file names based on datetimestamp_001, 002, 003 etc similar to current output tool

[Ned]

Good suggestion. That would be fairly straight forward to do as a batch macro if you wanted it sooner. Feel free to reach out to me if you wanted any help.

[chris_love]

Hi Adrian - I'd be happy to share my existing S3 / Redshift macro upload and as Ned says we can adapt for lots of files.

Have you tried the new Amazon Redshift Buk upload in the Output Data tool in v10? Keen to hear experiences if so.

[adrianloong]

Hi Everyone,

Thanks for the offer to help Chris. Ned has kindly created a workflow which then calls a batch s3 macro. I have attached it so that others can benefit

The s3 macro is that it takes fileNum as a input counter and then replace the filename with the latest counter

Replace([Destination], "XXX", PadLeft([#1], 3, '0'))

[Community_Admin]

The status of this idea has been changed to 'Inactive'. This status indicates that:

1. The idea has not had activity in the form of likes or comments in over a year.

2. The idea has not reached ten likes.

3. The idea is still in the 'New Idea' status. 

However, this doesn't mean your idea won't be implemented! The Community can still like and comment on this idea. With enough renewed interest, this idea can be brought back into the 'New Idea' status. 

Thank you for contributing to the Alteryx Community and the Alteryx Product Idea Boards!