Split output to multiple json files for S3 Upload

[christophjones]

I'd like to manage file size by splitting my output into multiple json files.

The solution presented here works great when writing locally or to a network drive:

https://community.alteryx.com/t5/Alteryx-Designer-Knowledge-Base/How-To-Split-Output-Into-Multiple-F...

However, when the output is changed to Amazon S3 Upload, the option to take filename from field is lost and the results write to a single file.  Any guidance is appreciated.

Thanks

[DavidP · Accepted solution]

Hi @christophjones 

I think the best way to do this is to write a batch macro with the S3 upload tool. Attached is an example.

I made it so that you enter the key, secret, bucket and object folder in a text input tool in the main workflow. The filename gets generated using the group ID field. Note that in the workflow calling the macro, I use the Group By function when calling the macro. This ensures that the macro only writes the relevant Group ID's rows to each file.

The only thing with this method is that the macro expects the data fields as defined in the macro input tool.

Let me know if you have any questions.

[christophjones]

Very slick - thanks for the quick response.