AWS S3 upload issues / Block Until Done woes

I have to develop a series of workflows that all create different files and then upload the output to AWS S3.

I thought I would be really smart and create a macro to accept the filename (that was just created from the workflow) and upload the file. That way the AWS credentials are all in one place - in the macro.

I've now developed the workflow that processes the data, writes the results to the CSV file and then passes the CSV filename to the macro.

I've used a "Block Until Done" to ensure that the file is written before the macro attempts to upload, I've also used the "Wait a Second" (even set it to 60) and the macro always runs before the file is created. I know that "Block Until Done" is only for records flowing through the tool, but how do I wait for the file to be created?

Workflow

Amazon S3

Best Practices

Accepted answers

cmcclellan

I've fixed the problem by refactoring the workflow like this:

Now it works all the time, I just have to fix the macro inputs not being saved properly.

All comments

cmcclellan

I've fixed the problem by refactoring the workflow like this:

Now it works all the time, I just have to fix the macro inputs not being saved properly.

philipmannering

Hey @cmcclellan. Was there a reason for not using the Amazon S3 Upload Tool?

cmcclellan

I am using the S3 Upload tool, but in the macro because I wanted the credentials in one place (ie not in every workflow) So the macro is basically "upload this file" and the different workflows create the file before calling the macro.

Bdogg

Could you explain why this approach works? It's not clear to me why it would be effective.

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703