cancel
Showing results for 
Search instead for 
Did you mean: 

Alteryx designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.

S3 Sub-Buckets

Meteor

OK... new to this, so I'd appreciate your assistance. Thanks in advanced...

 

The S3 Download tool allows designer to see all files in a specific S3 bucket. What do you do when your files are in S3 sub-buckets so the files are not directly under the bucket specified in the configuration tool (see example attached)? Something like this: //S3/bucket/state/cities.json

 

Thanks again,

 

Ernie C.S3_Bucket.JPGS3 Sub Bucket

 

Highlighted
Magnetar
Magnetar

Please see @DanC's excellent post, and ensuing discussion, here: https://community.alteryx.com/t5/Alteryx-Knowledge-Base/Amazon-S3-Download-Use-Wildcards-to-Select-a....  Hopefully it can get you where you need to be.

 

Meteor

Thank you, yes, I had read and implemented this solution and I'm able to read the contents of the S3 bucket. My challenge is that the contents in the S3 buckets is a series of folders, and the files I want are in every one of those folders. It's the drilldown into the folders that I'm unable to perform in order to extract the files in each of those folders.

 

Up to that point, the solution is awesome. Thanks for suggesting it.

Magnetar
Magnetar

Interesting problem here... I'm reviewing this stuff (AWS docs) and looking to create a listing of all objects using the Download tool. It's certainly possible since I was able to make the correct HTTP calls using Postman, however Postman automatically generates the required signatures; the trick is to create all that with Alteryx and then submit the appropriate signed HTTP request.  Anyway - so no new news, but thought I'd comment on what the ideas I'm thinking about.

Magnetar
Magnetar

HI @ecastruita,

 

This one has been bugging me, so I took some time to recreate the "Using GET with an Authorization Header" here, in R running in an Alteryx workflow, (attached).  There's a fair bit of R code in the R tool, but otherwise it's a nice, simple workflow...

 

zpic.png

 

In the beginning text field, you'll need to provide your:

  • host
  • endpoint
  • request_parameters
  • region
  • method
  • service
  • public key
  • secret key

... which sounds like a lot, but is straightforward if you're in S3 much.  From there, the workflow should list everything returned by the query thus defined; presumably from there you could use the usual Alteryx tools to parse objects of interest.

 

Hope that helps!

 - John

 

Meteor

Thanks john!! I'm playing around with it. What makes solutions complicated is working internally to Amazon, our environment is so much different than what Amazon AWS customers experience.

 

I'll give you a shout out if/when I can make it work :-).