community
cancel
Showing results for 
Search instead for 
Did you mean: 

alteryx server Discussions

Find answers, ask questions, and share expertise about Alteryx Server.
SOLVED

Stiching together multiple workflows? Crew Runners? Alteryx Gallery Connector? other?

Highlighted
Alteryx Partner

Hi All,

 

I am currently wrestling with the best approach in order to split a large/complex workflow & having it run off a server installation. To make the workflow functional/readable/manageable has resulted in it being broken into 6 separate workflows and in some of these I have had to write some of the datasets to a yxdb output in order to be ingested by a subsequent part.

 

Without server in the picture crew macro conditional runners look like the best approach & in fact I have this running in this manner off my local, however once server is involved and these sub flows are loaded into the Gallery 2 issues present themselves:

1) It doesn't appear to me that you can tell the runner to point at a workflow stored in the gallery (please correct me if I am wrong but this suggests that I am not - https://community.alteryx.com/t5/Alteryx-Server-Knowledge-Base/Chaining-Workflows-Together/ta-p/3857...)
2) The yxdb files I have written using the constant Engine.WorkflowDirectory but once this moves to server this directory would be dynamic for each run. Now I can work around this by introducing my own user constants which should address this by specifying either a network drive or somewhere local to the machine (please advise if you think there is a better approach).

 

One options to address the above which has been tossed around has been to stick master workflow with the crew runners into the gallery & have it scheduled as any other workflow but have the sub flows stored in a network drive / local drive. This I can see works but defeats the purpose of the gallery as now these subflows wouldn't be stored in the gallery for version control/access etc.

 

As a result of the above this sent me looking and thanks to some assistance from the Alteryx team have pointed me at the Gallery Connector: https://gallery.alteryx.com/#!app/Alteryx-Gallery-Connector/5ad07a4a8a933718049c73d2

 

To assist others if you are looking at this the install was a little cumbersome. I have downloaded other tools from the Gallery and installed them no issue but this one needed some lateral thinking as I couldn't get it to install. After downloading the file, double clicking the yxzp file, designers asks to import the package, which then results in opening a workflow which simply produces a google report with static text. Effectively it is the same text as seen on the screen in the gallery with a link to download the yxi file from here: https://alteryx-connectors.theinformationlab.co.uk/downloads/AlteryxGallery.yxi

 

Now when I click this link it doesn't download a yxi file it in fact downloads a zip file i.e. AlteryxGallery.zip. To install this you need to rename the zip file to have a yxi extension .... but I digress.

 

Now that I have it installed I see that the tool has no input connector only what appears to be an output connector.

 

So now I can't see how I can stitch my workflows together using this tool because I obviously would want it to be part 1-> part 2-> part3 -> part4->part5->part6. Am I missing something or is this not possible with this tool ?

 

Another approach I have been contemplating is given these workflows will only run once a day is to have a table in a DB or even a text file which effectively has the last run time of each of the parts. This value would be updated at the end of each parts run. I could then schedule each part to run say 10-15mins apart one after the other and the first step could be to read the status of the previous part and if it hasn't run today to not continue with the workflow but this seems to me like a bit of workaround as opposed to a proper solution.

 

Another approach which was bounced around has been to potentially have each part built as a macro. I haven't built my own macros before so I don't know if this is even possible or if it would require drastic change given how the parts have been built i.e. to write to temp files to be ingested further down or how to potentially pass these output as inputs to the subsequent macro.

 

These are the approaches I have come up with and am just weighing up which path to pursue. So have I missed any alternatives ? have others tried these and found some better than others ? Any assistance / guidance would be much appreciated.

 

Sorry for the long post.

 

 

Alteryx Partner

 

I would still be interested in hearing peoples views on solving this but decided to read and learn on the macro front and managed to convert my sub-flow parts into macro's and build a single flow (see pic below)

 

The only issue I see with this is on the testing front as you can't cache the DB calls if the whole flow takes a few minutes to run in this implementation you have to run the flow completely e2e. I do have other macro questions but i'll put them in the designer forum.

 

flow_as_macro.png

 

 

 

 

Alteryx Certified Partner

I think what you want is to Chain the workflows together.

 

Check this out, something i read i haven't actually gotten around to try it.

 

https://community.alteryx.com/t5/Alteryx-Server-Knowledge-Base/Chaining-Workflows-Together/ta-p/3857...

Alteryx Partner

Thanks @j_acon for the response .... I read through the linked post and it appears if you see the question posted underneath it the same point I have raised is still being put forth i.e. the approach works for locally hosted resources but once the gallery / server comes into the mix where all the flows are hosted there then the approach can't be used.

 

I am thinking the macro approach I have taken is currently the most workable approach to this. Looking at the Alteryx gallery connector since it only has and output connector then appears will only working if you want to run multiple workflows to produce dataset for you to use as inputs to your flow but you can't pass data between gallery hosted workflows.