Control Order of Execution of Workflow Objects (Container)

[s_pichaipillai]

Hi 

Wanted to control the order of execution of objects in Alteryx WF but right now we have ONLY block until done which is not right choice for so many cases 

Can we have a container (say Sequence Container) and put piece of logic in each container and have control by connecting each container?
Hope this way we can control the execution order
It may be something looks like below 

[pvigneron]

Hello,

We are getting close to 4 years 😉

[darryl5280]

When people want to control their containers, I think they should be sent in a good direction of converting their Containers to Macros.

On the left-click menu for Containers add a function called “Convert to a Macro”, which is only taking a combination of the Container’s Inputs and Outputs and converting them all to the “Macro Input” and “Macro Output” tools in a new workflow.

With all the Containers converted to a Macro, then all the other Alteryx Tools can be used with then to order the sequences and allowing all of them to logically flow together like the built in Alteryx Tools.

[EEdLoh]

@darryl5280 I've tried using macros to control tool execution order. It didn't work as expected. I think Alteryx reads the tools out of a macro then inserts them into the overall order of execution as it sees fit.

[DanH]

@EEdLoh @darryl5280 In order for the macro to fully block its interior processes, it has to be a batch or iterative macro, not a standard macro. Batch macros will guarantee each and all iterations are complete before passing any records to the output stream. With this method, even independent, unconnected streams within the macro will be fully complete before downstream processes start.

See https://community.alteryx.com/t5/Alteryx-Designer-Knowledge-Base/Controlling-the-Order-of-Operations... for a working example.

[KylieF]

Thank you for your post! We are really interested in this idea, however we cannot fit this idea on the near future road map. We’ll keep this idea in mind however and update the status once we’re better able to speak on when and if we can implement it.

[brianscott]

@KylieF - I know you are just the messenger here, but this thing is literally the highest voted idea in the ideas section with 207 votes.  This is the one the community is clamoring for and has been for the last forever.

Until we have this, Alteryx will never be able to serve as something resembling an orchestration tool; you will be able to create brilliant one offs, but for a large scale strategy of moving data from one place to the other when job A must complete before job B, it just can't be trusted to do sequential operations without join-hacks, or an ever increasing number of 'batch' macros in a russian doll box, or macro runner cascades.  

[KylieF]

@brianscott

I completely understand your frustration and I truly appreciate your input on this feature. While I understand this may not be the answer you are looking for, I have reached out to our product teams for a more detailed reasoning for this status change. At this point in time we decided moving this idea to the Revisit status would be the most accurate and transparent status. This is due to the primary team who would tackle this feature currently working through the E2 - Next Generation Engine Beta. However, once this feature is out of beta and has been released to the general public, we'll have the bandwidth to re-address this idea in it's entirety and provide a clear response and timeline for if we are able to pursue this idea.

I'll make sure update this idea again as soon as I have an new information on the status of this idea.

[brianscott]

@KylieF 

OK!  Please let the product team understand that there are situations wherein the 'solutions' provided earlier in this thread are simply not viable.  

1) Put in a batch macro!  If I need to run this this in gallery, then my macro cannot have a data connection in it, because data connections inside macros don't work in Gallery (also been on the idea list for the last forever).   So, if my order of execution requires outputting to a data source, *and* I need to have it run on gallery, I cannot enforce ordering by macro usage.  

2) Use a block until done and a fake join / fake append to 'force' the engine to wait!  Again, if our datastream from block 1 of a block until done ends in an output tool, there is nothing on the back side of that tool, so I cannot put my fake append datastream after output operation is done; i.e., I have no mechanism to force the datastream to wait until the outputting is completed.  

3) Use Crew Macros!  Works great so long as you do not want to have access to datastreams from one chunked workflow to another, but for many use cases (most) where you are trying to enforce an order of operations, you need data from earlier parts of the process.  I love crew macros, but if my data stream has important chunks of information in it, using conditional runners doesn't help.  

Thanks.

[KylieF]

@brianscott 

Thank you for the clarification, however, I do not believe your first point is accurate as data connections within a macro can be uploaded to the gallery and run successfully. Tool containers within a macro can also be used as a work-around, as tool containers allow for a greater control over the workflow process. 

We also have a Knowledge Base Article over this process.

While I understand that it would still be best for this feature to exist in the product rather then having to use workarounds, I wanted to insure any misconceptions on how the Gallery and data connections work was addressed.

Thank you,

KylieF

[C3PO]

Another orchestration type option to consider:

https://community.alteryx.com/t5/Engine-Works-Blog/Using-the-Alteryx-Gallery-API/ba-p/449007