Upload Multiple json files from Download tool into Amazon S3

Question

I have multiple api calls I'm making that return JSON data. I want to take the json responses from the download tool and import them directly into S3 as json files.

I have figured out how to save the json files to my hard drive by using a file name that has the drive path. I want to do this same process except upload it directly to S3 instead of my hard drive.

Is there a way to do this?

help.png

workflow.png

SRadford · Answer

I have researched batch macros and it looks like I have to use batch macros and AWS-CLI to get it to work. I still don't have it figured out, but I'm working on it. It's interesting that AWS upload tool doesn't have something built for this use case. Hopefully this is something they improve in the future.

jrgo · Answer

Hi @SRadford

I think you have the right idea, but you may need to place the S3 Upload tool into a Batch Macro using the File Name field as your Control Parameter which would update the "Object Name" field so that each row is stored individually.

If you're not familiar with creating a Batch macro, do a search on this community and you'll find tons of content to help you out.