ORDER OF EXECUTION OF COMPONENTS

[chinu267]

Good Day. If you look at the attached workflow, then it should run in order from top to bottom.

Rather what is happening is the below components of top container executes first:

Then it straight goes to the bottom most container to execute it completely. The problem here is the step before this has not run. The file in last step uses the old input and this messes up what I expected it to do. 

Why Alteryx executes on its own? Does it help? Why simple things are so complicated? Made for a purpose?

I expected it to run from top to bottom and all these days it has been running on its own

[Emil_Kos]

Hi,

Please look into the block till done tool. I think it should help you.

https://help.alteryx.com/current/designer/block-until-done-tool

[chinu267]

block until done will not help for what I am trying to do.

[chinu267]

Did a block until done but still one of the files in first container gets written last

[CharlieS · Accepted solution]

---

@chinu267 wrote:

Why Alteryx executes on its own? Does it help? Why simple things are so complicated? Made for a purpose?

I expected it to run from top to bottom and all these days it has been running on its own

---

There is logic built into Alteryx to optimize execution within the Alteryx engine. The issue here is that there's nothing to tell Alteryx that one container should be run before the other. Where you place tools on the canvas is not a parameter the engine uses. If there are just two input tools on the canvas with no other connections, then Alteryx doesn't know that there's a priority to one over the other.

It does not automatically make the connection that you're writing to one source and then reading it because there are too many read/write sources/methods to control for (blocking vs non-blocking, files vs databases, replace vs append vs sequential, etc). 

So my question is this: can we do all the steps in a single run? I get why these would be separate containers when you're designing the process, but I don't see a programmatic reason why you need to save a file and read it again to start another process when you could just stream one into the other. In the attached example, I remove any Input that is written by an Output in the same workflow and connected the data streams directly. (I may have made a mistake, but hopefully the idea is clear)

Let me know if this could work.

[chinu267]

Thanks for looking into the problem and providing a solution. 

Yes we can do all the steps in a single run and will follow your guidelines. Only thing I did it insteps is due to the fact tht business uses those intermediate files to verify/validate when anything goes wrong or for general auditing.

Appreciate your valuable time and suggestions. Have a good weekend ahead.

[Michael_Draper]

I have use cases where the situation you present is actually preferable to joining everything together. Essentially, a large process that is modular, and the pieces are often, but not always, run together. So by outputting an intermediate data set and reading it in, the two pieces can be run independently or together by enabling/disabling a container without any need to reconfigure additional steps. This is especially helpful if the first piece is particularly time consuming, as is often the case.

Since this has been an open issue/request for a long time, I am assuming that what feels to the user like a simple ordering (don't input this tool until this other specific output tool has finished) is actually fairly troublesome on the back end. "Block Until Done" works if everything is along the same data stream, but that is not always the case, unless you do some convoluted workarounds. (Maybe something like an interface tool that can connect with anything with the action tool interface as a more "global" block until done would be useful?)

It seems to me, the best existing solution here is to make the containers separate batch macros, and call them from a "control" workflow with dummy parameters passed to the macros. That, still, is workaround. It would be nice if there were a more native, intuitive way to call other workflows in order, even if it were just native tools that did that exact workaround in the background.

For example, a "Call Workflow" tool with an optional input and an output. If it is connected on its input, it will wait to begin execution until it receives data. The output would be the log file. The configuration would just be a directory to point the workflow at. I think this could all be created through a batch macro right now, but would be incredibly helpful if Alteryx had worked out the particulars and made it easy for users to manage and order multiple workflows.

Would love your thoughts on this, @Charlie S. I will post in the ideas section, as well. Thanks!

[CharlieS]

@Michael_DraperI agree, sometimes it is necessary to split the process into intermediary steps. Often I'll do this when working with a large amount of data as to manage machine resources and distribute work appropriately.

In your case, I'd recommend checking out the Runner macros (specifically, Conditional Runner) from the CReW macros. If you build your steps as separate workflows and utilize relative referencing, those macros should do the trick.

http://www.chaosreignswithin.com/p/macros.html 

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Crew-Macro-Conditional-Runner/m-p/4913... 

[Michael_Draper]

That's a good point, I suppose I really just described the runner macros. I wonder if those will ever be added into the native program instead of as an add-on, but that is probably the ideal solution at this point. Thanks!

[Hipponoceros]

it is extremely disappointing that there is not some sort of solution to this yet!