Community Spring Cleaning week is here! Join your fellow Maveryx in digging through your old posts and marking comments on them as solved. Learn more here!

Alteryx Server Knowledge Base

Definitive answers from Server experts.

Migrating Workflows

jperrotto
Alteryx
Alteryx
Created

[Note:  An OAuth2 version of the workflow described below for Server 2021.4+ can be found in the Gallery, https://community.alteryx.com/t5/Community-Gallery/Alteryx-Gallery-Workflow-Migration/ta-p/1031739 ]

A frequent feature request we hear from customers is the ability to do true siloed environments when developing and testing workflows. Alteryx Server already has a versioning feature which allows you to publish a specific revision of a workflow, however this can lead to confusion and potentially incorrect versions being run when a large number of workflows and users exist in a system.

To make versioning of workflows easier we have now included functionality that allows for migrating workflows between multiple Alteryx Server environments.

In Alteryx Server 2018.4 you can now manage separate environments and deploy a workflow from one environment to the next. This allows for completely siloed environments, and gives customers greater ability to manage the lifecyle of their workflows.

How does it work?

Within the admin portal of Alteryx Server, there is now a new option under the Workflow section to enable Workflow Migrations. This global flag will enable a set of controls at the workflow level to mark a workflow as "Ready for migration". Workflows can be marked as ready for migration in one of two places.

1. In the admin portal, under the Workflows section. Within a workflow, you will see an option to mark the workflow as "This workflow is ready to be migrated".
2. Within your private Studio under the "Workflow Settings" dialog for a specific workflow.

The core functionality of the migration is handled in four new endpoints. Three of the endpoints will be in your source environment (where workflows will be migrated from) and one of the endpoints exists in the target environment (where you will be publishing workflows to). All of these endpoints authenticate using our Oauth1 auth scheme just like all of our other public admin API endpoints.

These four endpoints are:

[Source Environment]
[HTTP GET]
[api/admin/v1/workflows/migratable/?subscriptionIds={comma separated subscriptionIds}/]
1. The first endpoint exposes an array of ApplicationIds which have been marked as ready to migrate. This endpoint takes a comma separated list of subscriptionIds as a query parameter and will return all workflows marked as ready to migrate under the specific studio(s). If no subscriptionsIds are provided then all workflows that have been marked as ready to migrate should be returned. This is a lightweight endpoint that will return just three properties. The appId, the currently published revision ID, as well as the subscriptionID the workflow belongs to.

[Source Environment]
[HTTP GET]
[api/admin/v1/{appID}/package/]
2. The second endpoint will download the actual workflow as a .YXZP. This endpoint takes an appID as a parameter and downloads the entire workflow as a package.

[Target Environment]
[HTTP POST]
[api/admin/v1/workflows/]
3. The third endpoint is a publish endpoint. This allows you to take the workflow obtained in the second endpoint and publish to the target environment. This post uses a multipart form-data request. The form sections and properties can be found in the code example.

[Source Environment]
[HTTP PUT]
[api/admin/v1/workflows/migratable/{appID}/]
4. The fourth and final endpoint allows you to toggle the "ready to migrate" flag on a given workflow back to false in the source environment after the workflow has been successfuly published in the target environment.

The flow of these migration will typically follow the below:

Get list of apps ready to migrate ->
Loop through list and for each app returned ->
Download individual workflow from source environment ->
Publish workflow in target environment ->
Reset "ready to migrate" flag on workflow in source environment

This functionality was built into API endpoints in order to give the customer ultimate flexibility in designing their own migration process. An example C# script is attached to this post to help demonstrate how a simple migration could occur. This code is just a sample however, and should not be used for your own migration process.

 

***Attached is also a proof of concept of using these new endpoints. It is not supported, but just an example of how to use the new APIs! You will need to enter the target and source consumer keys and secrets (marked by red annotations) in order for this to work. And it will only migrate workflows that have been marked ready for migration***


Note: Alteryx Server versions 2021.3 and above are impacted by Defect (TGAL-6268) which fails to retrieve GET requests to /admin/v1/workflows without search parameter. Please use "Gallery Workflow Migration v2.zip" for 2021.3 and above. 

Changes made to OAUTH Macro are as follows:
 

[Source Environment]
[HTTP GET] [api/admin/v1/workflows/] API calls have been replaced with [HTTP GET] [api/admin/v1/workflows/all/]

Since the JSON response body is different from the original workflows API Call, The "download workflow metainfo container" has been changed to accommodate for API change. 
 

  • Filter tool (ID 56) has been modified to retrieve new field names
  • Select tool (ID 79) added to change the naming convention back to expected results
  • Formula tool (ID 80) to add expected workerTag field name. note: this field will not be populated with any results on Target server.



Additional Resources

Attachments
Comments
NatSnook
8 - Asteroid

This is a fantastic bit of code! am so glad i found it. Hopefully we can get it working will save a lot of governance headaches! one question when the workflow is uploaded to the target gallery is there a way we can modify the workflow options? ie user must specify credentials? seems like when it's uploaded its possibly expecting to not require any credentials?

dokmanc
6 - Meteoroid

I am using the program.cs .NET example for promoting code. I modified it to update it with my source and target url, my source keys and my target keys. When I run the code and trace through it, I see that it is returning the items marked as "Ready to Migrate" but when I get an "Unauthorized" error in the PublishAppInTargetEnvironment on the line "client.UploadData(url, httpBody);" even though I am a curator in both the source and target environments.


When I switch the source and destination, the same thing occurs. the code returns a collection of the items ready for release but I get the same error when trying to migrate in the other direction.

 

 

 

bdaunt
6 - Meteoroid

I am also getting an Unauthorized response (401) on the  PublishAppInTargetEnvironment upload. Using the exact code in the Program.cs with the BaseUrl, APIKeys and Secrets changed to reflect our servers. Both of which I am a Curator in and both of which I can use all of the other Admin level methods without a problem.

nnjithendra7
5 - Atom

Hi,

Did any one got the solution regarding source id parameter issue. Where if we publish the workflow using API end point it is creating duplicate even if we specify the source id.

Thank you,

Jithendra.

 

Rahul_Thakur
5 - Atom

@nnjithendra :- Are you passing the same source ID each time? If no then duplication will occur

nnjithendra7
5 - Atom

Hi Thakur,

Yes, I am passing the same source ID each time. But it is creating duplicate once. I.e. Initially I am publishing workflow with out source ID and the workflow got published successfully. Next time if I am publishing the same workflow, I am passing the respective workflow id. Then it is creating duplicate, where it has to overwrite the existing workflow. If I try to publish the same workflow again (for couple of more times), then it is not creating duplicate. I.e. each workflow it is creating at least one duplicate workflow, after the it is working as expected.

If I remove one of the duplicate workflow from Gallery and try to publish again the same workflow, then it will create duplicate and it will not create duplicates for subsequent publish.

 

Thank you,

Jithendra.

David-Carnes
12 - Quasar

@nnjithendra7 
The "source ID" is a bit of a misnomer; it has nothing to do with the unique identifier that gets generated the first time it's uploaded.  You can put anything into it, like "Hello World" for example.

 

You need to publish it via the API with the Source ID already filled in.  Subsequent publishes via the API will overwrite the original.  It will not create an entry in version control, however.

Yongcan
8 - Asteroid

Currently we upgrade gallery to 2021.3 and see some enhancement/issue regarding the version control once published via API.

 

Under 2021.3, once a workflow is migrated to target gallery with Admin API. it's not simply override as version 1.

Now the first version is version 0 , and once published via Admin API, a new version created on target gallery. ( which is good as now we start having multiple versions on target)

 

However we notice some issue:

1. we update workflow meta info with each migrate and notice the meta info is not populate via the latest/published version. ( people from other studio see published version workflow's meta info)

2. Since we see multiple version on target gallery, we try choose old version as published, however once done we hit some defect or something as the page won't load and give a blank screen so we have to migrate the workflow again, this actually make the new feature not working at all. 

 

We are working with alteryx support to understand if it's defect or anything wrong.

 

 

xAJBx
7 - Meteor

I am having a hard time understanding how the endpoint /admin/v1/{appId}/package/ downloads the corresponding workflow from the target server.

 

I have successfully hit the endpoint on the api document example page and with a JS Express app.  It looks like the response Body is the raw compressed alteryx workflow.

 

If anybody can give me guidance on how I write this down to disk I would be more than grateful, I'm sure I am over looking something here. 

 

Attached are some images to help visualize what I struggling with.

Server_Test_api_admin_v1_appid_package.PNGJS_call_to_alteryx_apia_with_axios.PNGConsole_log_of_axios_request.PNG

alishadhillon__
8 - Asteroid

Whilst it is possible to migrate workflows from server to server, does anybody know if you can also migrate workflows from collection 1 to collection to, both within the same server?

 

If this is a yes, is it also done with the server's admin API calls?

rgottumukkala
5 - Atom

Hi

 

When i try to migrate workflows using "Gallery Workflow Migration v2.zip" ,i am getting below error:, can you please let me know what am i misssing.

 

"ErrorLink: Join (30): https://community.alteryx.com/t5/*/*/ta-p/710269?utm_source=designer&utm_medium=resultsgrid|The field "id" is missing. Compare the tool configuration with the input stream."

 

rgottumukkala_0-1661195699526.png

 

connection info:

rgottumukkala_1-1661195738885.png

 

Thanks

Raju

lepome
Alteryx Alumni (Retired)

@rgottumukkala

It appears that there are no data going into that Join.  Figure out why that is first. 

rgottumukkala
5 - Atom

Hi , Do this Workflow works in 2021.4 version?, we are getting id mismatched error

Yongcan
8 - Asteroid

@alishadhillon__ 

 

Moving around workflow from Collection now support under Admin V3 API.

 

POST /v3/collections/{collectionId}/workflows
Add a workflow to a Collection

 

DELETE /v3/collections/{collectionId}/workflows/{appId}
Remove a workflow from a Collection

EdP
Alteryx
Alteryx

For Server 2021.4+, an OAuth2 version of the example Migration workflow can be found in the Alteryx Community Gallery: 

 

https://community.alteryx.com/t5/Community-Gallery/Alteryx-Gallery-Workflow-Migration/ta-p/1031739

rymatt830
7 - Meteor

Edit: also tagging @lepome because she seems to know everything!

 

@EdP @jperrotto I'm having trouble getting the new Gallery Workflow Migration workflow (for 2022.1 and above) to perform migrations. It appears to be breaking with the Join (293) tool in 6.1 Create New Workflow. That join tool has an incomming right input but no incomming left input, and as a result, the J anchor output doesn't contain any data. Has anyone else run into this issue? Is this a mistake in the workflow? See the image below.

 

Join (293) tool with no left inputJoin (293) tool with no left input