Organizing Workflows

Question

I'm wondering if anyone with more experience with server has a recommendation on using file names to organize workflows within a collection.

I was thinking of something like

[Team Name] - [Workflow Type] - [Workflow Short Description]

Not having a ton of experience with workflows, I can't think of what would be different types that would be relevant. I was hoping that someone else has done something similar and might have some good categories for workflow types?

Maybe:

DataIngestion

TableBuild

DataExport

...?

Any ideas are welcome.

hroderick · Answer

Here is the CreW macro source http://www.chaosreignswithin.com/p/macros.html

I like them because they let me organize the execution of many small workflows instead of having all the steps in one large workflow.

Crew macros make it easy to orchestrate their execution in proper order.

They a provide nice success/fail outcome path to next step that is visualized instead of hidden in an event configuration.

A lot of our workflows use Run Command tool to run command line programs. The Run Command configuration and connections are so convoluted that I like putting it in a workflow almost by itself. This creates more workflows to orchestrate.

I think Gallery running would be OK if you had many end users or small teams independently running and scheduling their own process. At my work becoming scheduled crosses the team threshold to become an enterprise operation with higher performance, centralized control, and reliability expected. Trying to keep over a hundred scheduled tasks and serving dozens of user communities from a single gallery collection or studio would be maddening. In present state (2020.1) the gallery gui just doesn't handle an enterprise number of objects very well.

We keep the workflows and apps and supporting assets on an Alteryx server in windows directories and run them without using Gallery scheduling. They are organized in subdirectories for supporting team / job /  workflows, bat file, other assets

We need the workflows in Gallery only so the Alteryx Connect will catalog them. Before Connect, the workflows would come to me to be scheduled as email attachments. We changed the handoff to be done by developer saving to their Gallery private collection, then sharing with a collection named Prod Stage, then informing me it was ready to be received. From the collection I download to prod server, import to its windows directory, write a bat file, then schedule the bat file . The change in process assures the latest version is in Gallery for Connect to catalog.

SideOfRanch · Answer

So is this how your organize your files more locally? I'm not familiar with Crew macros but the little bit I can google says it's tools and utilities that can be used locally. So you are are primarily talking about your local file organization and then you do something to bring it together to then publish to a collection?

hroderick · Answer

You are on the right track putting heirarchy n the name because they get all mixed together in studios and collectitons.

I generally write small functional workflows and tie them together with CreW macros.

The last part of the name indicates function:

*Driver is the controlling process with CreW macro

*Extract pulls data from sources and does basic cleaning then writes to yxdb

*Process brings all the cleaned yxdb's together and is usually the most complex then writes output files

*Check does any balancing or other quality assurance checks

*Update saves results to database tables

*Refresh outputs to Tableau datasources

The * in all the names are all the same and describe the purpose of the workflow or app. I keep them in a subdirectory named *. I keep copies in the gallery but keep the master in a file. The subdirectory contains all other assets along with workflows.