Data repository for recurring worflows

Question

Dear all,

Hope you are doing well.

I want to add a repository for the status of data every time I run my workflow. I was thinking of creating a .yxdb at the end of the workflow and using that as my input/output onwards. For the ensuing times I run it, I was thinking of "resetting" the status at the beginning of the workflow and then update the column based on the result before the end.

I have tried it but keep getting errors all over the place, and I have started to overcomplicate things. Keep thinking there has to be an easier way of doing this.

Thank you for your help or ideas!

Juan1

danilang · Accepted Answer

Hi @juan1

The trickiest part about reading from and writing to the same file is synchronizing so the two operations occur in the order you want and that you don't have the read blocking the write or vice versa.  This process is complicated be the fact that the Alteryx engine runs the tools in the order that they're added to the canvas.  Use the following workflow as an example

If you add the Input Data tool to the canvas before the Output data tool(InBeforeOut.yxmd), this workflow runs as expected.  If you added the Output first and then the Input(OutBeforeIn.yxmd), the Output tool starts to write to the repo file before the Input can finish reading it giving the dreaded "File in use by another process" error.

If you need to guarantee a read-change-write order, you can use a blocking tool to ensure that everything is complete before the output starts writing.  In the following(repo.yxzp) I added a Summarize after the union that just groups by all the fields in the repo file. Since the data can't flow through the summarize until all the records are grouped, this ensures that the Input tool completes before the Output tool can start.

Dan