Migrating Workflows

[Note: An OAuth2 version of the workflow described below for Server 2021.4+ can be found in the Gallery, https://community.alteryx.com/t5/Community-Gallery/Alteryx-Gallery-Workflow-Migration/ta-p/1031739 ]

A frequent feature request we hear from customers is the ability to do true siloed environments when developing and testing workflows. Alteryx Server already has a versioning feature which allows you to publish a specific revision of a workflow, however this can lead to confusion and potentially incorrect versions being run when a large number of workflows and users exist in a system.

To make versioning of workflows easier we have now included functionality that allows for migrating workflows between multiple Alteryx Server environments.

In Alteryx Server 2018.4 you can now manage separate environments and deploy a workflow from one environment to the next. This allows for completely siloed environments, and gives customers greater ability to manage the lifecyle of their workflows.

How does it work?

Within the admin portal of Alteryx Server, there is now a new option under the Workflow section to enable Workflow Migrations. This global flag will enable a set of controls at the workflow level to mark a workflow as "Ready for migration". Workflows can be marked as ready for migration in one of two places.

1. In the admin portal, under the Workflows section. Within a workflow, you will see an option to mark the workflow as "This workflow is ready to be migrated".
2. Within your private Studio under the "Workflow Settings" dialog for a specific workflow.

The core functionality of the migration is handled in four new endpoints. Three of the endpoints will be in your source environment (where workflows will be migrated from) and one of the endpoints exists in the target environment (where you will be publishing workflows to). All of these endpoints authenticate using our Oauth1 auth scheme just like all of our other public admin API endpoints.

These four endpoints are:

[Source Environment]
[HTTP GET]
[api/admin/v1/workflows/migratable/?subscriptionIds={comma separated subscriptionIds}/]
1. The first endpoint exposes an array of ApplicationIds which have been marked as ready to migrate. This endpoint takes a comma separated list of subscriptionIds as a query parameter and will return all workflows marked as ready to migrate under the specific studio(s). If no subscriptionsIds are provided then all workflows that have been marked as ready to migrate should be returned. This is a lightweight endpoint that will return just three properties. The appId, the currently published revision ID, as well as the subscriptionID the workflow belongs to.

[Source Environment]
[HTTP GET]
[api/admin/v1/{appID}/package/]
2. The second endpoint will download the actual workflow as a .YXZP. This endpoint takes an appID as a parameter and downloads the entire workflow as a package.

[Target Environment]
[HTTP POST]
[api/admin/v1/workflows/]
3. The third endpoint is a publish endpoint. This allows you to take the workflow obtained in the second endpoint and publish to the target environment. This post uses a multipart form-data request. The form sections and properties can be found in the code example.

[Source Environment]
[HTTP PUT]
[api/admin/v1/workflows/migratable/{appID}/]
4. The fourth and final endpoint allows you to toggle the "ready to migrate" flag on a given workflow back to false in the source environment after the workflow has been successfuly published in the target environment.

The flow of these migration will typically follow the below:

Get list of apps ready to migrate ->
Loop through list and for each app returned ->
Download individual workflow from source environment ->
Publish workflow in target environment ->
Reset "ready to migrate" flag on workflow in source environment

This functionality was built into API endpoints in order to give the customer ultimate flexibility in designing their own migration process. An example C# script is attached to this post to help demonstrate how a simple migration could occur. This code is just a sample however, and should not be used for your own migration process.

***Attached is also a proof of concept of using these new endpoints. It is not supported, but just an example of how to use the new APIs! You will need to enter the target and source consumer keys and secrets (marked by red annotations) in order for this to work. And it will only migrate workflows that have been marked ready for migration***

Note: Alteryx Server versions 2021.3 and above are impacted by Defect (TGAL-6268) which fails to retrieve GET requests to /admin/v1/workflows without search parameter. Please use "Gallery Workflow Migration v2.zip" for 2021.3 and above.

Changes made to OAUTH Macro are as follows:

[Source Environment]
[HTTP GET] [api/admin/v1/workflows/] API calls have been replaced with [HTTP GET] [api/admin/v1/workflows/all/]

Since the JSON response body is different from the original workflows API Call, The "download workflow metainfo container" has been changed to accommodate for API change.

Filter tool (ID 56) has been modified to retrieve new field names
Select tool (ID 79) added to change the naming convention back to expected results
Formula tool (ID 80) to add expected workerTag field name. note: this field will not be populated with any results on Target server.

Additional Resources

Server API Overview

Gallery Workflow Migration v2.zip

Gallery Workflow Migration.zip

Workflow

Accepted answers

All comments

michael_treadwell

This is fantastic! I have a few questions.

First, I've upgraded to 2018.4 but I do not see these endpoints in the interactive documentation: https://gallery.alteryx.com/api-docs/

Are there plans to add them?

Second, even though this was intended to migrate between environments, could it also be used to migrate workflows within environments? For example, if I have a workflow on Gallery and I want to switch the workflow owner from one private studio to another, could I do that with the workflow POST endpoint? I noticed that the form has an "owner" section.

jperrotto

Hi @michael_treadwell,

Thanks for the feedback, below I'll try to answer both of the questions that you have:

1) We were mindful of not putting the new endpoints on this page for right now, since they don't have to do with a single server install and more to do with multiple installs and we didn't want to confuse anyone with putting them in the same place. That being said, we might add them somewhere else on that page going forward.

2) Yes, it's entirely possible to use the new endpoints with your server but it would essentially "copy" the workflow over to a new studio. Note that any schedules that were pointed to the old workflow will not know anything about the new workflow.

Let me know if you have any further questions.

Thanks!

Jeff

sameerpremji

Hi @jperrotto,

We have had a controller/scheduler component (version 2018.4) running on a Windows 7 workstation, while we were piloting the product.

We recently purchased an Alteryx Server and are in the midst of setting it up. Once the Alteryx Server is up and running, how do I transfer all the workflows that are stored and scheduled in the controller into the Alteryx Server?

Is there a quick migration tool I could use?

Thanks!

jperrotto

Hi @sameerpremji,

Thanks for reaching out to me. As long as the schedules from Alteryx Designer were setup to run in the Scheduler DB (see image below), there is a way to migrate them to over to Alteryx Server...

You will either need to have admin access to your server or have someone else set this up, but here are the steps. Click on the Jobs navigation tab on the left, then click on the Migrate tab (as long as schedules are enabled on server), you should see these schedules in the list and you can migrate them onto your new server. Once you migrate them, there is no undoing, so make sure you select the correct person to own the schedules.

Thanks!
Jeff

sameerpremji

Hi @jperrotto,

Correct me if I'm wrong but based on your above response, I get the impression that you're suggesting a way of migrating workflows from one server to another. This is not that case in my situation.

There is no "Gallery Admin Portal" on the Windows 7 workstation, so I can't browse and migrate the scheduled jobs/workflows by navigating to 'Jobs' section on a browser. That workstation only contained Alteryx Designer with Scheduler install. So, how can I migrate the jobs from the workstation (without any server component) to an Alteryx Server running a Web portal?

Also, on another note, I couldn't find anywhere in setup/install/configure documentation as to whether to first install a Web Server (IIS or Apache) on an Alteryx Server first or does the Alteryx Server setup utility installs the Web Server for you automatically or not?

jperrotto

Hi @sameerpremji,

It sounds like you have a workstation with Designer and schedules on it , you're setting up an Alteryx Server right now, and you want to migrate those old schedules over to the new Alteryx Server? We should be able to help you get these schedules migrated as long as they were setup with that first option to run in the DB and not on disk. I suggest you reach out to Alteryx Support at support@alteryx.com. They should be able to answer all of your questions there and walk you through this process more clearly.

Also, to answer your other question, Alteryx Server does not run on Apache or IIS, it starts itself up.

Thanks,
Jeff

sameerpremji

Hi @jperrotto,

That is correct -- yes, all of our schedules were setup with that first option to run in the DB and not on disk. I'll contact Support as you've suggested.

So, the Alteryx Server software automatically opens up Ports 80 and 443 and starts serving web pages in the Admin Web Portal?

Do you have a link to the instructions on how to setup SSL Certificate so that we can hit the Admin Portal via HTTPS (Port 443)?

Thanks a lot

jperrotto

Hi @sameerpremji,

Yes, once the server configuration process is complete it should automatically start up a new service called AlteryxService, that will be hosted on port 40 (by default).

Here is an article that one of our customer support team members wrote about SSL:

https://community.alteryx.com/t5/Alteryx-Server-Knowledge-Base/Configuring-Alteryx-Server-for-SSL-Obtaining-and-Installing/ta-p/15577

Feel free to bring up any other questions you may have with our support team.

Thanks,

Jeff

sameerpremji

Thanks a lot @jperrotto for your help.

levell_x_dunn

@jperrotto,

This is pretty cool. I have a ways before I can actual try this out, but I'm assuming there is a way just like you set the workertag and canDownload, to set the run modes.

I'm thinking of changing the default to safe mode on our Private gallery and only ones that we have migrated over are set to unrestricted mode.

Any thoughts or suggestions? Good idea, bad idea?

jperrotto

Hi @levell_x_dunn,

We actually do not have a run mode option when migrating workflows, this is definitely something we can look into for a future update though. Nice suggestion! Currently I believe all the workflows are migrated over with the run mode that is set in the Server configurations. If you wanted to do this right now, there would be a lot of manual work involved; every time you migrate a workflow over to the new Server you would have to change the run mode settings on that workflow.

Thanks!
Jeff

jsoler

@JeffreyP,

I have a question regarding the last step of the solution about migrating worflows from one environment (Let's say sandbox) to production (let's say Live).

when the process calls the API to PUT (publish the workflow in the new environment), the process actually copy the workflow on the new enviorement, what happens if we have an old version of that workflow on the destination enviroment? we would have a second copy or a new version will populates on the destination?

Also, another question about the functionality of Migrate on: Gallery Admin>Jobs>Migrate after Schedule Workflow on my computer checking the option "Run a copy of the worflow stored in the Scheduler DB". Is this option another way to do the migration? which is the best option? the API option (that you explain at the beginning of this post) or this second option?

Thanks a lot for your time,

jperrotto

Hi @jsoler,

Thanks for reaching out to me. You are correct, by default this will create a new workflow in your destination environment, so if you already had one, there will now be two of them now. However, with this API you can provide a 'sourceId' (the workflow id in the destination environment) and then it will overwrite the existing workflow with the new one that you copied over.

Similarly, yes. You could use the Scheduler option in Designer and point to the Live environment to copy it over. The thought here is that you'll have to manually do each workflow one at a time via Designer. And with the API it would be possible to write code such that it would copy as many workflows as you wanted to.

Thanks,

Jeff

sameerpremji

Hi @jperrotto ,

We have an Alteryx Gallery Server on-premise and we have a few folks with Alteryx Designer who publish workflows onto the Gallery.

An Analyst typically creates and designs a workflow and saves this file locally on their PC. Once they're happy, the publish it to the on-premise Gallery. However, if there are some enhancement to be made, the Analyst opens the same locally stored file and makes the necessary changes.

When they publish the same workflow file onto the Gallery, it does not prompt us that "A workflow with this name already exists. Do you want to overwrite?". Instead, it actually publishes the enhanced workflow file, although with the same name, as a new workflow. This results in us cleaning up duplicates as well as re-configuring specific permissions for collections, users, etc.

The GUI method does not allow us to "overwrite" the same workflow file, which I think only encourages duplicates and extra-work. We also use Tableau Server and when Tableau Publishers use Tableau Desktop to publish their workbooks, the server actually detects the same workbook name and asks/prompts the publisher/author whether to overwrite or create a totally new workbook.

I have not had a need to use the API method as we mainly use the Designer tool to publish our workflows onto our Gallery.

Any advise? We also have created this enhancement request as an idea and we're hoping someone will notice it.

jperrotto

Hi @sameerpremji ,

My recommendation would be to actually have your users open the workflow in Designer from the Gallery, rather than from their local computer. The Gallery has no knowledge of local files on your computer. And you are correct, file names are not unique on the Gallery, this was done by design.

The steps I would take here; open up Designer, go to File -> Open Workflow -> Select your Gallery -> Select the workflow. Then after you make your changes/edits to the workflow, go to File -> Save, and this will treat it as a new version of that workflow, instead of treating it as a Save As for a new workflow. Essentially treating the Gallery as your file repository instead of your local computers, since you also have version control in the Gallery, you can set which version is "published" (ie. which version your viewers will be executing on the Gallery).

Thanks,
Jeff

AmeliaG

@jperrotto Any recommendations on which code compiler to use?

jperrotto

Hi @AmeliaG ,

It's really to the users discretion. As long as it supports multi-part forms. But I would recommend using whatever they're comfortable with.

Thanks,

Jeff

PeterA

Hi @AmeliaG

Here is an export version as an Alteryx Macro. Yes it can be done without R or Python with just the native tools within designer. However, if you are like me with some self-signed certs you are probably going to run into a problem. So, this is a quick python based macro that would allow you to override the SSL verification process.

#################################
from ayx import Package
from ayx import Alteryx


#################################
import requests, urllib, hmac, hashlib
import collections, base64, math, random, string, time, sys, os
import pandas as pd


#################################
params = Alteryx.read('params')
params = params.iloc[0]
oauth_consumer_key = params['oauth_consumer_key']
oauth_consumer_secret = params['oauth_consumer_secret']
gallery_migrate_out = params['gallery_migrate_out']
request_proto = 'https://' if params['gallery_migrate_out_SSL'] > 0 else 'http://'
ssl_verify = False if params['gallery_migrate_out_self_signed_SSL'] > 0 else True
ayx_stage_directory = os.path.join(os.path.normpath(params['workflow_stage_directory']), '')



#################################
class Gallery(object):
    def __init__(self, apiLocation, apiKey, apiSecret):
        self.apiLocation = request_proto + apiLocation + '/gallery/api/admin/v1'
        self.apiKey = apiKey
        self.apiSecret = apiSecret


    """Generate pseudo-random number"""

    def generate_nonce(self, length=11):
        return ''.join([str(random.choice(string.ascii_uppercase + string.digits + string.ascii_lowercase)) for i in
                        range(length)])


    """Returns HMAC-SHA1 signature"""

    def generateSignature(self, httpMethod, url, params):
        q = lambda x: requests.utils.quote(x, safe="~")
        sorted_params = collections.OrderedDict(sorted(params.items()))

        normalized_params = urllib.parse.urlencode(sorted_params)
        base_string = "&".join((httpMethod.upper(), q(url), q(normalized_params)))

        # Python 3 requires string in bytes for hmac.new()
        secret_bytes = bytes("&".join([self.apiSecret, '']), 'ascii')
        base_bytes = bytes(base_string, 'ascii')
        sig = hmac.new(secret_bytes, base_bytes, hashlib.sha1)

        # Python 3 requires use of b64encode method from base64
        return base64.b64encode(sig.digest())


    """Constructs OAuth Parameters"""

    def buildOAuthParams(self):
        return {'oauth_consumer_key': self.apiKey,
                'oauth_signature_method': 'HMAC-SHA1',
                'oauth_timestamp': str(int(math.floor(time.time()))),
                'oauth_nonce': self.generate_nonce(11),
                'oauth_version': '1.0'}


    """Finds workflows that have been flagged for migration"""

    def migrate (self):
        method = 'GET'
        url = self.apiLocation + '/workflows/migratable/'
        params = self.buildOAuthParams()
        signature = self.generateSignature(method, url, params)
        params.update({'oauth_signature': signature})

        try:
            output = requests.get(url, params=params, verify=ssl_verify)
            output.raise_for_status()
        except requests.exceptions.HTTPError as err:
            print(err)
            sys.exit(err)
        except requests.exceptions.RequestException as err2:
            print(err2)
            sys.exit(err2)

        return output, output.json()


    """Exports the App that was requested"""

    def getApp(self, appId):
        method = 'GET'
        url = self.apiLocation + '/' + appId + '/package/'
        params = self.buildOAuthParams()
        signature = self.generateSignature(method, url, params)
        params.update({'oauth_signature': signature})

        try:
            local_filename = ayx_stage_directory + appId + '.yxzp'
            # NOTE the stream=True parameter below
            with requests.get(url, stream=True, params=params, verify=ssl_verify) as yxzp:
                yxzp.raise_for_status()
                with open(local_filename, 'wb') as f:
                    for chunk in yxzp.iter_content(chunk_size=8192):
                        if chunk: # filter out keep-alive new chunks
                            f.write(chunk)
                            # f.flush() 

        except requests.exceptions.HTTPError as err:
            print(err)
            sys.exit(err)
        except requests.exceptions.RequestException as err2:
            print(err2)
            sys.exit(err2)
        return






#################################
con = Gallery(gallery_migrate_out, oauth_consumer_key, oauth_consumer_secret)
response, data = con.migrate()
migratableWorkflows = response.json()
df = pd.DataFrame (migratableWorkflows)
df.rename(columns={'id':'appId'}, inplace=True)
df['fullPath'] = ayx_stage_directory.replace("/","\\") + df['appId']  +'.yxzp'
for row in df.itertuples():
    pkg = con.getApp(row.appId)



#################################
Alteryx.write(df,1)

jdm5650

@PeterA is it possible to download this macro from somewhere? I set up a Sandbox and Production environment at my company, but have been struggling to figure out how to enable migrating workflows from one environment to another.

neeleshapatil1

I agree. @PeterA could you please if possible make this macro available on the community site? Also does this just migrate apps (.yxwz) files? Hopefully any resources available on gallery like yxmd, yxmc or yxwz files can be downloaded and pushed to target environment.

SeanAdams

If anyone is struggling to get the workflow uploader up and running in Postman - here are the steps to get this working:

Step 1: Get a simple call working with oAuth

First get connected to your server using OAuth authentication, with the appropriate keys - and confirm that you're getting a good connection using oAuth Authentication on the GET method for workflows.

- Add in the API URL for workflow list

you can find this on <your gallery>\api-docs. for example: http:\\localhost\gallery\api-docs)
On my local server, the workflow endpoint is http://localhost/gallery/api/admin/v1/workflows/

Set the Authorization on the auth tab

use oAuth 1
You need the admin key and the admin secret (blurred out below) which you get from the gallery admin
all the rest leave as default

Postman - 1 - good call to server.png

You can tell you're getting a good response because of the highlighted "200" server status; and the response in the body which shows some of your workflows.

Step 2: Change this to an upload call

Set up the POST query using exactly the same OAuth credentials and URL for the API
Use the exact same API endpoint - \workflows. For example: http://localhost/gallery/api/admin/v1/workflows/
Then change the body to "form-data"
Set the first key to be "file" and hover over the end to change type from "text" to "file" and select your packaged file to upload
Add in the remaining form components per the screenshot or per @jperrotto 's C# code attached above
Then you should get a good response from the server - it will return your new AppID

Open Question: We're still experimenting with the SourceID to see if we can get it to override the canvas instead of duplicating a new one - but no-luck so-far.

cc: @adams_ca ; @revathi ; @Kosi @Sotoll ; @PeterA

adams_ca

Thank @SeanAdams !

I have found that if you specify the sourceId for a workflow that already exists, even though the expected behavior is that the workflow will be updated/overwritten, it is in fact not. A new workflow is always created using this API.

Any ideas/solutions on how to ensure that a new workflow isn't published to the gallery when that workflow already exists?

PeterA

Hi @adams_ca,

I suspect something may be off with the body in your gallery POST call - as it should overwrite the workflow. I've found that spaces in the appID and/or not getting the case correct for my keys (e.g. must be "sourceId" not "SourceID) will however generate a new workflow.

Now this may be a good thing or a bad thing. What happens is when you do get the sourceId correct it will overwrite the workflow, and unfortunately setting latest version is 1 & published version is 1 - with no previous history is preserved. @JeffreyP, is there a method to get versioning to work?

jperrotto

Hi @PeterA ,

When you migrate a workflow from one server to another and you supply a sourceId, it's assumed that you want to replace that workflow, not add a new version of that workflow. It's the same as the replace functionality that's currently on the Gallery. This was by design. So currently there is no way to add versions of the same workflow. The idea behind it was that you'd already do this in the target environment and when you promote the workflow to the source environment, you'd already know the version that you were promoting and wouldn't need to keep track of it there. But we're always open to discussions if you think this is something that is widely wanted.

Thanks,
Jeff

SeanAdams

Thank you @jperrotto - this is definitely necessary - even from a regulatory and roll-back perspective, we have to be able to keep versions. The regulatory angle is that under SOX we need to keep an active audit trail of changes to prod assets. The roll-back perspective is that you need the ability to roll-back to a previously known good version.

So yes - the best behavior would be to reversion the asset with the provided App ID so that each version is kept along with an audit trail.

piyush

Did anyone implement the solution Jeffrey provide in their organisation?

jdm5650

To anyone still looking for a solution around how to use these endpoints, Michael Treadwell (@michael_treadwell) had a session on this at Inspire 2019 and posted a link on the Interworks blog with macros that you can download and use to 1. pull all workflows that are ready to be migrated in a source environment and 2. publish those workflows to a target environment and reset the migration ready flags in the source environment. After months of looking for an answer, I was finally about to set this up for my company's environment this morning because of Michael's session and resources. Hopefully me sharing this here will help others.

https://interworks.com/blog/mtreadwell/2019/06/12/the-migratory-patterns-of-the-common-alteryx-workflow/

(Personal note on implementation: Make sure you pay attention to the part at the end where he says to make sure all the studios involved are paid type and have an expiration date. This hung me up for an hour or so today.)

*Edited intial post to correctly tag Michael

neeleshapatil1

While running Gallery Migration Workflow uploaded by JeffreyP received following error. Could you please confirm if there is any specific .exe file required to be placed and what would be it’s path?

Error: PublishWorkflow (18): Record #1: Tool #1: Failed to run external program "..\Supporting_Macros\AlteryxGalleryAPI_PublishWorkflow.exe": The system cannot find the file specified.

I also see as per below screenshot that command arguments has some hard coded values like gallery server etc. How it gets updated relevant to our sever details?

jperrotto

Hi @neeleshapatil1 ,

If you downloaded the attached workflow, you will need to make sure that the executable file (AlteryxGalleryAPI_PublishWorkflow.exe) is a part of the package. Since the run command tool is inside the macro, these hard-coded values will be updated when it runs. The only things you should have to update are the server key/secret and the url for the source and target servers in the parent workflow.

Thanks!
Jeff

neeleshapatil1

Thanks JeffreyP for your quick reply.

But where i can find this executable file? Might me my company network preventing .exe file to be downloaded along with all other macros.

Also additional to this would like to know on the following point.

There could be multiple private studio’s under source Alteryx server, we would like to get workflows from a particular private studio to be migrated to same private studio in the Target server. How this can be achieved?

As an Gallery admin, I can only see API key and secret of the private studio that I am part of on the source environment. How I can get API key and secret for other private studios so that we can migrate workflow from a particular private studio to same private studio in the Target server.

Quick Links

This months top contributors

atcodedog05 19598

Qiu 15878

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703