Programmatically/Manually Create .yxzp

[PWD5032]

Background:

I'm attempting to automatically deploy Alteryx solutions to Alteryx Server via Azure DevOps CICD.  In order to do that in the way I need it, I have to figure out how to take a folder full of yxmc and yxmd files and programmatically turn that into a yxzp.

Main:

We know that yxzp files are a compressed binary format.  If you change the extension from yxzp to zip and unpack, you'll find the constituent workflows.  The challenge that I'm hoping the community can assist me with is finding a means of doing the opposite.

Unfortunately it doesn't seem as simple as compressing a file or collection of files and changing the extension to zip.

Quick nomenclature abbreviations that I'll be using…

p-yxzp: properly created yxzp using the export option in Alteryx

c-yxzp: custom yxzp created by manually compressing one or more workflows/macros then changing the extension to yxzp

Here are my observations:

• Practical loading - loading a yxzp into Alteryx
• p-yxzp: after unpacking Alteryx knows to open the primary workflow of that yxzp
• c-yxzp: Alteryx unpacks the file but does not open any of the constituent workflows
• Xml file differences
• The primary workflow of the yxzp has additional code added to it (in red) describing encoding and MetaInfo of the outputs of any connectors in the workflow: (See the attached images)
•  
• 
• 

• When tested with a parent workflow that calls a child workflow, I found that the child workflows and macros are unchanged

I tried copying the xml from the resulting primary workflow of the unpacked p-yxzp and using that code in a workflow where I then manually zipped it up and changed the extension but I didn't get the desired result upon opening in Alteryx.

Packaging a yxzp using Alteryx must be doing something to indicate what the primary/parent workflow is but I can't tell what the indicator is.  I feel like figuring this part out would crack the problem.

So, does anybody have any thoughts?

Thank you!

-Patrick

[patrick_digan]

@PWD5032 I just saw your post (a few months late I know). I'm not sure exactly how @tlarsen7572 solved it, but in this post I pointed out that the p-yxzp has a comment field in the properties that contains the name of the workflow that should open up. 

[tlarsen7572]

If I remember right, I just added/filled in the comment field in the yxzp's file properties with the name of the workflow, as @patrick_digan mentioned, and everything worked.

[DavidM]

Hey guys,

Just wanted to share this...

I am helping build some CI/CD pipelines on delivery of workflows/projects under GIT repo into Servers programmatically through migration APIs. This requires YXZP format to make the API call.

The pseudo-code below worked with Python to take my workflow folder with YXMD and turn it into YXZP.

Ultimately the YXZP requires the COMMENT to be in place as this post mentions. This must reflect the name of the main YXMD.

#Here goes the bad boy

from zipfile import ZipFile

import os

from os.path import basename

# create a ZipFile object

dirName = "/Users/dmatyas/Workflow"

with ZipFile('Workflow.yxzp', 'w') as zipObj:

   # Iterate over all the files in directory

   for folderName, subfolders, filenames in os.walk(dirName):

       for filename in filenames:

           #create complete filepath of file in directory

           filePath = os.path.join(folderName, filename)

           # Add file to zip

           zipObj.write(filePath, basename(filePath))

           zipObj.comment = b'Lesson 3 Exercises.yxmd’

[DavidM]

Hi guys,

I have a more robust solution than my previous post (which works but is obviously more a pseudo-code than anything else).

Attached is a YXZPBuilder package that delivers on packaging YXZP in a most robust fashion.

Note: CREDITs due to the Connect loaders team here at AYX.

What this bad boy can do? There are 3 methods in the class

1/ First one generates the YXZP package, and you can specify which subfolders should be packaged into yxzp

2/ Second can scan all yxmds, yxmcs, etc and change their AYX version in their headers (if you need to work across multi version deployment for whatever reasons)

3/ Can update metainfo (probably not super relevant, but can help if working with WFs metadata)

Here is a sample usage

from loaders_yxzp_builder import YXZPBuilder
YXZPBuilder.create_package(r"c:\\Temp\\Python\\SandBox", "sample_workflow.yxmd", "my_data.csv", "External resources")

[David-Carnes]

@DavidM 

Thank you for teaching me Python.  Seriously.  I've been dissecting the code for a project of mine and I learned a lot.  However, I do have some issues with the code.

Two things:
1.  The yxzp files weren't actually getting compressed.  Opening them in Notepad++ revealed all the XML, prefaced and suffixed with binary code.  Adding zipfile.ZIP_DEFLATED inside create_package(*args) fixed that.

with zipfile.ZipFile(yxzp_full_path, 'w', zipfile.ZIP_DEFLATED) as current_package:

 2. OK, here's the showstopper.  The comment added to the compressed yxzp file works great, in that Alteryx Designer knows which file to open after importation.  It works with yxzp files of one workflow or workflows with supporting macros.

But when attempting to publish the newly minted yxzp to Alteryx Server, only compressed files with no internal directory structure will succeed.   (Using Postman to POST to the /api/admin/v1/workflows/ endpoint.)

First, there's this in the logs:

There was an unknown error executing the application. The engine reported status Error

Then if a run is attempted, this:

Problem Loading App
Could not create Directory "C:\ProgramData\Alteryx\Service\Staging\XProcessCache\F82B7A2A82F54356\_externals/1/"

If I take the same workflow and export it from Designer, the file is the same size but there are no issues with either publishing via Postman or running the app in Gallery.

I'm at a loss on what is missing from or wrong with the output of the Python code.

I hope you can provide some guidance.

Thank you very much for your time.

Sincerely,

David

[DavidM]

Hi @David-Carnes,

Thanks for that post.

Did you try to explore the YXZP builder package that I have shared later on after that initial post?

It may fix the issues you listed.

david

[David-Carnes]

Hello @DavidM !
Yes, I have been through both sets of code and used both the Python tool in Designer and PyCharm.  I used the latter to step through the code.  But the problem is not the creation of the yxzp but the error when publishing via a POST to the /api/admin/v1/workflows/ endpoint.  It goes in and I get the unique ID back but the logs show a failure after that.

I did some further playing and discovered that the server is OK with one layer of subdirectories but it barks at any subdirectories nested within the first layer.  My solution will be to create some code to copy the nested directories to the root then delete the _externals folder.  This does necessitate a pass through the affected workflows, apps, and macros to correct the paths to the content in the subdirectories.  But I'm already doing something like this to swap out old data sources for shiny new Gallery Data Connections in the destination server.

(Off topic, I had written my "replace old data sources with new" in R, using readLines() and gsub().  Worked great in R Studio; the R tool chokes on files longer than a thousand or so.  I might try it in Python.)

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/R-Tool-readLines-function-fails-with-q... 

Thank you for posting the code; it was illuminating.

Ciao,

David

[Thableaus]

Hey @David-Carnes did you end up finding a good solution for this nested directories problem?

[David-Carnes]

Yes, I created a macro to pull all files out of the _externals/[some number] and put them in the root directory with the workflow YXMD or app YXWZ..

OK, some context.  This is the heart of an app I created to allow my users to migrate the workflows over to the new server.  It had to be manual because I wanted to move to an Active Directory authentication.

To speak to the nested directories problem, I created a macro to move the files in _externals, delete the _externals directory, and update the XML of the YXDB or YXWZ.

Attached are the remove_externals_dir macro and a supporting dynamic directory macro.

Hope this helps!

Ciao,

David