AWS S3 upload issues / Block Until Done woes

[cmcclellan]

I have to develop a series of workflows that all create different files and then upload the output to AWS S3.

I thought I would be really smart and create a macro to accept the filename (that was just created from the workflow) and upload the file. That way the AWS credentials are all in one place - in the macro.

I've now developed the workflow that processes the data, writes the results to the CSV file and then passes the CSV filename to the macro.

I've used a "Block Until Done" to ensure that the file is written before the macro attempts to upload, I've also used the "Wait a Second" (even set it to 60) and the macro always runs before the file is created.  I know that "Block Until Done" is only for records flowing through the tool, but how do I wait for the file to be created?

[cmcclellan · Accepted solution]

I've fixed the problem by refactoring the workflow like this:

Now it works all the time, I just have to fix the macro inputs not being saved properly.

[PhilipMannering]

Hey @cmcclellan. Was there a reason for not using the Amazon S3 Upload Tool?

[cmcclellan]

I am using the S3 Upload tool, but in the macro because I wanted the credentials in one place (ie not in every workflow)  So the macro is basically "upload this file" and the different workflows create the file before calling the macro.

[Bdogg]

Could you explain why this approach works? It's not clear to me why it would be effective.