I'm attempting to automatically deploy Alteryx solutions to Alteryx Server via Azure DevOps CICD. In order to do that in the way I need it, I have to figure out how to take a folder full of yxmc and yxmd files and programmatically turn that into a yxzp.
We know that yxzp files are a compressed binary format. If you change the extension from yxzp to zip and unpack, you'll find the constituent workflows. The challenge that I'm hoping the community can assist me with is finding a means of doing the opposite.
Unfortunately it doesn't seem as simple as compressing a file or collection of files and changing the extension to zip.
Quick nomenclature abbreviations that I'll be using…
p-yxzp: properly created yxzp using the export option in Alteryx
c-yxzp: custom yxzp created by manually compressing one or more workflows/macros then changing the extension to yxzp
Here are my observations:
I tried copying the xml from the resulting primary workflow of the unpacked p-yxzp and using that code in a workflow where I then manually zipped it up and changed the extension but I didn't get the desired result upon opening in Alteryx.
Packaging a yxzp using Alteryx must be doing something to indicate what the primary/parent workflow is but I can't tell what the indicator is. I feel like figuring this part out would crack the problem.
So, does anybody have any thoughts?
Just wanted to share this...
I am helping build some CI/CD pipelines on delivery of workflows/projects under GIT repo into Servers programmatically through migration APIs. This requires YXZP format to make the API call.
I have a more robust solution than my previous post (which works but is obviously more a pseudo-code than anything else).
Attached is a YXZPBuilder package that delivers on packaging YXZP in a most robust fashion.
Note: CREDITs due to the Connect loaders team here at AYX.
What this bad boy can do? There are 3 methods in the class
1/ First one generates the YXZP package, and you can specify which subfolders should be packaged into yxzp
2/ Second can scan all yxmds, yxmcs, etc and change their AYX version in their headers (if you need to work across multi version deployment for whatever reasons)
3/ Can update metainfo (probably not super relevant, but can help if working with WFs metadata)
Here is a sample usage
from loaders_yxzp_builder import YXZPBuilder YXZPBuilder.create_package(r"c:\\Temp\\Python\\SandBox", "sample_workflow.yxmd", "my_data.csv", "External resources")
Thank you for teaching me Python. Seriously. I've been dissecting the code for a project of mine and I learned a lot. However, I do have some issues with the code.
1. The yxzp files weren't actually getting compressed. Opening them in Notepad++ revealed all the XML, prefaced and suffixed with binary code. Adding zipfile.ZIP_DEFLATED inside create_package(*args) fixed that.
with zipfile.ZipFile(yxzp_full_path, 'w', zipfile.ZIP_DEFLATED) as current_package:
2. OK, here's the showstopper. The comment added to the compressed yxzp file works great, in that Alteryx Designer knows which file to open after importation. It works with yxzp files of one workflow or workflows with supporting macros.
But when attempting to publish the newly minted yxzp to Alteryx Server, only compressed files with no internal directory structure will succeed. (Using Postman to POST to the /api/admin/v1/workflows/ endpoint.)
First, there's this in the logs:
There was an unknown error executing the application. The engine reported status Error
Then if a run is attempted, this:
Problem Loading App Could not create Directory "C:\ProgramData\Alteryx\Service\Staging\XProcessCache\F82B7A2A82F54356\_externals/1/"
If I take the same workflow and export it from Designer, the file is the same size but there are no issues with either publishing via Postman or running the app in Gallery.
I'm at a loss on what is missing from or wrong with the output of the Python code.
I hope you can provide some guidance.
Thank you very much for your time.
Thanks for that post.
Did you try to explore the YXZP builder package that I have shared later on after that initial post?
It may fix the issues you listed.
Hello @DavidM !
Yes, I have been through both sets of code and used both the Python tool in Designer and PyCharm. I used the latter to step through the code. But the problem is not the creation of the yxzp but the error when publishing via a POST to the /api/admin/v1/workflows/ endpoint. It goes in and I get the unique ID back but the logs show a failure after that.
I did some further playing and discovered that the server is OK with one layer of subdirectories but it barks at any subdirectories nested within the first layer. My solution will be to create some code to copy the nested directories to the root then delete the _externals folder. This does necessitate a pass through the affected workflows, apps, and macros to correct the paths to the content in the subdirectories. But I'm already doing something like this to swap out old data sources for shiny new Gallery Data Connections in the destination server.
(Off topic, I had written my "replace old data sources with new" in R, using readLines() and gsub(). Worked great in R Studio; the R tool chokes on files longer than a thousand or so. I might try it in Python.)
Thank you for posting the code; it was illuminating.
Yes, I created a macro to pull all files out of the _externals/[some number] and put them in the root directory with the workflow YXMD or app YXWZ..
OK, some context. This is the heart of an app I created to allow my users to migrate the workflows over to the new server. It had to be manual because I wanted to move to an Active Directory authentication.
To speak to the nested directories problem, I created a macro to move the files in _externals, delete the _externals directory, and update the XML of the YXDB or YXWZ.
Attached are the remove_externals_dir macro and a supporting dynamic directory macro.
Hope this helps!