S3 Sub-Buckets

[ecastruita]

OK... new to this, so I'd appreciate your assistance. Thanks in advanced...

The S3 Download tool allows designer to see all files in a specific S3 bucket. What do you do when your files are in S3 sub-buckets so the files are not directly under the bucket specified in the configuration tool (see example attached)? Something like this: //S3/bucket/state/cities.json

Thanks again,

Ernie C.S3 Sub Bucket

[JohnJPS]

Please see @DanC's excellent post, and ensuing discussion, here: https://community.alteryx.com/t5/Alteryx-Knowledge-Base/Amazon-S3-Download-Use-Wildcards-to-Select-a....  Hopefully it can get you where you need to be.

[ecastruita]

Thank you, yes, I had read and implemented this solution and I'm able to read the contents of the S3 bucket. My challenge is that the contents in the S3 buckets is a series of folders, and the files I want are in every one of those folders. It's the drilldown into the folders that I'm unable to perform in order to extract the files in each of those folders.

Up to that point, the solution is awesome. Thanks for suggesting it.

[JohnJPS]

Interesting problem here... I'm reviewing this stuff (AWS docs) and looking to create a listing of all objects using the Download tool. It's certainly possible since I was able to make the correct HTTP calls using Postman, however Postman automatically generates the required signatures; the trick is to create all that with Alteryx and then submit the appropriate signed HTTP request.  Anyway - so no new news, but thought I'd comment on what the ideas I'm thinking about.

[JohnJPS]

HI @ecastruita,

This one has been bugging me, so I took some time to recreate the "Using GET with an Authorization Header" here, in R running in an Alteryx workflow, (attached).  There's a fair bit of R code in the R tool, but otherwise it's a nice, simple workflow...

In the beginning text field, you'll need to provide your:

• host
• endpoint
• request_parameters
• region
• method
• service
• public key
• secret key

... which sounds like a lot, but is straightforward if you're in S3 much.  From there, the workflow should list everything returned by the query thus defined; presumably from there you could use the usual Alteryx tools to parse objects of interest.

Hope that helps!

 - John

[ecastruita]

Thanks john!! I'm playing around with it. What makes solutions complicated is working internally to Amazon, our environment is so much different than what Amazon AWS customers experience.

I'll give you a shout out if/when I can make it work :-).