Alteryx Server Discussions

Find answers, ask questions, and share expertise about Alteryx Server.
Don't forget to submit your entry for the Excellence Awards by October 30! | Need more information about the program? Check out the blog here
SOLVED

Stiching together multiple workflows? Crew Runners? Alteryx Gallery Connector? other?

Highlighted
Alteryx Partner

Hi All,

 

I am currently wrestling with the best approach in order to split a large/complex workflow & having it run off a server installation. To make the workflow functional/readable/manageable has resulted in it being broken into 6 separate workflows and in some of these I have had to write some of the datasets to a yxdb output in order to be ingested by a subsequent part.

 

Without server in the picture crew macro conditional runners look like the best approach & in fact I have this running in this manner off my local, however once server is involved and these sub flows are loaded into the Gallery 2 issues present themselves:

1) It doesn't appear to me that you can tell the runner to point at a workflow stored in the gallery (please correct me if I am wrong but this suggests that I am not - https://community.alteryx.com/t5/Alteryx-Server-Knowledge-Base/Chaining-Workflows-Together/ta-p/3857...)
2) The yxdb files I have written using the constant Engine.WorkflowDirectory but once this moves to server this directory would be dynamic for each run. Now I can work around this by introducing my own user constants which should address this by specifying either a network drive or somewhere local to the machine (please advise if you think there is a better approach).

 

One options to address the above which has been tossed around has been to stick master workflow with the crew runners into the gallery & have it scheduled as any other workflow but have the sub flows stored in a network drive / local drive. This I can see works but defeats the purpose of the gallery as now these subflows wouldn't be stored in the gallery for version control/access etc.

 

As a result of the above this sent me looking and thanks to some assistance from the Alteryx team have pointed me at the Gallery Connector: https://gallery.alteryx.com/#!app/Alteryx-Gallery-Connector/5ad07a4a8a933718049c73d2

 

To assist others if you are looking at this the install was a little cumbersome. I have downloaded other tools from the Gallery and installed them no issue but this one needed some lateral thinking as I couldn't get it to install. After downloading the file, double clicking the yxzp file, designers asks to import the package, which then results in opening a workflow which simply produces a google report with static text. Effectively it is the same text as seen on the screen in the gallery with a link to download the yxi file from here: https://alteryx-connectors.theinformationlab.co.uk/downloads/AlteryxGallery.yxi

 

Now when I click this link it doesn't download a yxi file it in fact downloads a zip file i.e. AlteryxGallery.zip. To install this you need to rename the zip file to have a yxi extension .... but I digress.

 

Now that I have it installed I see that the tool has no input connector only what appears to be an output connector.

 

So now I can't see how I can stitch my workflows together using this tool because I obviously would want it to be part 1-> part 2-> part3 -> part4->part5->part6. Am I missing something or is this not possible with this tool ?

 

Another approach I have been contemplating is given these workflows will only run once a day is to have a table in a DB or even a text file which effectively has the last run time of each of the parts. This value would be updated at the end of each parts run. I could then schedule each part to run say 10-15mins apart one after the other and the first step could be to read the status of the previous part and if it hasn't run today to not continue with the workflow but this seems to me like a bit of workaround as opposed to a proper solution.

 

Another approach which was bounced around has been to potentially have each part built as a macro. I haven't built my own macros before so I don't know if this is even possible or if it would require drastic change given how the parts have been built i.e. to write to temp files to be ingested further down or how to potentially pass these output as inputs to the subsequent macro.

 

These are the approaches I have come up with and am just weighing up which path to pursue. So have I missed any alternatives ? have others tried these and found some better than others ? Any assistance / guidance would be much appreciated.

 

Sorry for the long post.

 

 

Highlighted
Alteryx Partner

 

I would still be interested in hearing peoples views on solving this but decided to read and learn on the macro front and managed to convert my sub-flow parts into macro's and build a single flow (see pic below)

 

The only issue I see with this is on the testing front as you can't cache the DB calls if the whole flow takes a few minutes to run in this implementation you have to run the flow completely e2e. I do have other macro questions but i'll put them in the designer forum.

 

flow_as_macro.png

 

 

 

 

Highlighted
Alteryx Certified Partner

I think what you want is to Chain the workflows together.

 

Check this out, something i read i haven't actually gotten around to try it.

 

https://community.alteryx.com/t5/Alteryx-Server-Knowledge-Base/Chaining-Workflows-Together/ta-p/3857...

Highlighted
Alteryx Partner

Thanks @j_acon for the response .... I read through the linked post and it appears if you see the question posted underneath it the same point I have raised is still being put forth i.e. the approach works for locally hosted resources but once the gallery / server comes into the mix where all the flows are hosted there then the approach can't be used.

 

I am thinking the macro approach I have taken is currently the most workable approach to this. Looking at the Alteryx gallery connector since it only has and output connector then appears will only working if you want to run multiple workflows to produce dataset for you to use as inputs to your flow but you can't pass data between gallery hosted workflows.

 

 

Highlighted
7 - Meteor

I can't believe it is this difficult to run another workflow from the server, but in fact it seems to be so.  I too am on the server, and have a similar need.

 

I have a scheduled workflow that is essentially listening for a file (every day we have a batch job that publishes a trigger file).  My workflow reads this file and the last modified date and checks to see if the main workflow that uses this data has been run.  

 

I.e. if the batch trigger file is newer than the 'last workflow run date' then I want it to execute the primary workflow which will email the results to the users.  I am using the CReW macro 'Blocking Test' to kill my workflow if the batch file is not yet loaded.  My problem is I cannot seem to "call" the primary macro from the server.  From my local machine I could use the Conditional Runner or Command line to call the other workflow using the Alteryx.Engine.exe which will not work from the server.

 

This seems like functionality that should be a slam dunk on the server which should enable MORE power than is available on my desktop.  Don't get me wrong I get that it does in many ways (scheduling, sharing, etc) but as it comes to this functionality the server cannot seem to do what I need it to do which is frustrating!  Why doesn't Alteryx support this type of functionality on the server?

 

I cannot "chain" it because I do not want it to run.  I cannot command or Runner it because Alteryx.Engine.exe is on my C drive not the servers.  There should be an easier Alteryx supported tool for this!