Workflow Versioning

[ricoo]

Hi All,

My team and I are working on multiple workflows. We have a BASE folder where in the latest modifications of the workflows are stored. If we have a new project (for a new client - eg. ClientA), we copy all the latest workflows from the BASE folder to a new ClientA folder and modify the copied ones according to the business needs of ClientA. In a lot of cases, these new rules/logic we've applied for ClientA must be applied to the BASE workflows. My Colleague 1 is working on ClientB and I'm working on ClientC. That means the 3 of us are working on 3 different projects/clients. Which means every now and then, the changes we made to these specific client workflows must be merged to the BASE workflow. Our problem is, is there a way to systematically apply the changes to the client-specific workflows to the BASE workflows, and can still keep the different versions of it? Currently we're doing it manually. If I've made modifications for ClientC, I always need to copy it to the BASE. Then I must announce to the team that the BASE workflow has been updated so that they'll have to copy that recently modified workflow. It's supposed to be easy, but there are lot of times that this doesn't happen because we're just doing lots of projects at the same time. And if we have updated a workflow, we forget to copy it to the BASE and forget to tell the team about the update.

PS. It's a holiday season, this is my last day of work for this year. But if anyone can reply to this concern, my apologies in advance if I couldn't reply immediately. But I'll still check this every now and then during the holidays. Your help is very much appreciated. Thank you in advance and Happy Holidays.

[KGT]

A couple things that you can use to make this easier.

• You can create a workflow that parses the yxmd's to see what the differences are.
• Options > Advanced > Compare Workflows will allow you to visually see the difference.
  
  

When updating a template, it's always good to apply proper change management (or somewhere on the spectrum of proper). We have a few projects similar to yours whereby we copy the workflows and then work on that in a client folder. I have a planner (could be trello etc) board where team members can list the things that need to change. The other option is to move the parts that are most likely to change into a macro or a different workflow so that can be updated easily. 

Another thing we do if working on a client where we're adjusting the last models used for that client is building out changes in a separate workflow and then one person being in charge of implementing them with the required tests back into the base.

If you use a shared macro in one of these workflows, then when it is updated, it will update for wherever that is used. Be careful, as that would include already delivered projects. One of our core macros is stored as the current version, and every couple of months we'll roll additional changes into it and create a new version to use going forward.

Overall, this is a process question, but there are ways to set up your workflows to make them easier to create a robust process around.

[ricoo]

Hi @KGT ,

Thank you so much for replying on my question.

I have some clarifications/responses:

1. create a workflow that parses yxmd --what do you mean by this? I tried to google it, and found that there's an aftermarket tool that needs to be downloaded. Is this what you mean --Workflow XML Parser Tool?

https://help.alteryx.com/current/en/designer/tools/parse/workflow-xml-parser-tool.html#workflow-outp...

2. Visually comparing workflows via Compare Workflows for us is out of the picture, since we’re naming our alteryx workflows with a version number as our suffix (eg. Load Product 01, Load Product 02…) where in the latest version number is stored in the base, and the previous ones are stored in the archive folder. We also describe our changes inside the workflow using a comment tool so that anyone who opens it have an idea what changes were made

3. I’m kind of inclined to your idea of using macro, but I still need to think how to apply this to our set-up. I’m gonna need to do a brainstorming with my team mates by January when everyone is back. 

[apathetichell · Accepted solution]

Alteryx thinks it's biggest issue is something like 'cloud developement' or 'preparing a way to harness agentic processes' - it's not. it's versioning. It's always been versioning. It needs real github integration with the development process so you can push/pull edits to your workflow and maintain versions in a repo.

[KGT]

Hi @ricoo,

Alteryx Workflows are XML underneath and so that can be parsed to programmatically find the differences. You can connect to a YXMD in the input tool and read it in as either XML or CSV/flat with no delimiter. I always read it in as text and then parse it, but up to you.

This would be the workflow parser I would start with, but I find that I always do further customisation (actually, it's the same as your link, this is just the way to download it): https://marketplace.alteryx.com/en-US/apps/444941/workflow-xml-parser

You can also just parse a folder of workflows to get the comment boxes for instance.

@apathetichell, yep, I kind of agree. But what survives over time is not necessarily the product that is best for the job. Most are getting corralled into only using cloud data sources at their work. So, when IT goes to sign off on it, if it's on-prem they just say no in a lot of cases. I've seen architects do a hit job on Alteryx because it's not cloud and so makes their job harder. Inevitably, the user then has to struggle with another product that a cloud vendor "included", but the use cases are too far away from that decision maker. And most users can't explain how just having the control to do what they want is beneficial. The cloud conversation really annoys me, but I also prefer native apps rather than trying to work in a browser without access to the rest of my computer. I'm fine with needing things on cloud, but not at the expense of my efficiency.

[ricoo]

Hi @apathetichell @KGT ,

Again, thank you guys for your responses. I think @apathetichell is right, the closest solution would be to use a repository like GitHub. I've also asked a friend of mine, and he also told me to use GitHub for this kind of setup.

[apathetichell]

@KGT---> we could talk about this extensively - and I don't really disagree. My frustration can be when the same methodology is applied to Alteryx vs other tools which tbh have the same tech limitations but maybe market there workarounds/systems in a more palatable way -> and when the rebuilt python/pyspark version of the process doesn't have thorough data/UAT testing or encompass the same level of outliers.  Having said that - versioning is a core issue and -> checking code into/out of github for any final workflows/macros would be awesome and help control what you're running both locally and on Server.

[KGT]

Yep, Github is an option for storing the workflows. I've known several companies that did this.

If you're going that route, then I advise at least taking a look at these: https://marketplace.alteryx.com/en-US/apps/418313/github-tools