community
cancel
Showing results for 
Search instead for 
Did you mean: 

alteryx server Knowledge Base

Definitive answers from Server experts.

Migrating Workflows

Alteryx
Alteryx
Created on

A frequent feature request we hear from customers is the ability to do true siloed environments when developing and testing workflows. Alteryx Server already has a versioning feature which allows you to publish a specific revision of a workflow, however this can lead to confusion and potentially incorrect versions being run when a large number of workflows and users exist in a system.

To make versioning of workflows easier we have now included functionality that allows for migrating workflows between multiple Alteryx Server environments.

In Alteryx Server 2018.4 you can now manage separate environments and deploy a workflow from one environment to the next. This allows for completely siloed environments, and gives customers greater ability to manage the lifecyle of their workflows.

How does it work?

Within the admin portal of Alteryx Server, there is now a new option under the Workflow section to enable Workflow Migrations. This global flag will enable a set of controls at the workflow level to mark a workflow as "Ready for migration". Workflows can be marked as ready for migration in one of two places.

1. In the admin portal, under the Workflows section. Within a workflow, you will see an option to mark the workflow as "This workflow is ready to be migrated".
2. Within your private Studio under the "Workflow Settings" dialog for a specific workflow.

The core functionality of the migration is handled in four new endpoints. Three of the endpoints will be in your source environment (where workflows will be migrated from) and one of the endpoints exists in the target environment (where you will be publishing workflows to). All of these endpoints authenticate using our Oauth1 auth scheme just like all of our other public admin API endpoints.

These four endpoints are:

[Source Environment]
[HTTP GET]
[api/admin/v1/workflows/migratable/?subscriptionIds={comma separated subscriptionIds}/]
1. The first endpoint exposes an array of ApplicationIds which have been marked as ready to migrate. This endpoint takes a comma separated list of subscriptionIds as a query parameter and will return all workflows marked as ready to migrate under the specific studio(s). If no subscriptionsIds are provided then all workflows that have been marked as ready to migrate should be returned. This is a lightweight endpoint that will return just three properties. The appId, the currently published revision ID, as well as the subscriptionID the workflow belongs to.

[Source Environment]
[HTTP GET]
[api/admin/v1/{appID}/package/]
2. The second endpoint will download the actual workflow as a .YXZP. This endpoint takes an appID as a parameter and downloads the entire workflow as a package.

[Target Environment]
[HTTP POST]
[api/admin/v1/workflows/]
3. The third endpoint is a publish endpoint. This allows you to take the workflow obtained in the second endpoint and publish to the target environment. This post uses a multipart form-data request. The form sections and properties can be found in the code example.

[Source Environment]
[HTTP PUT]
[api/admin/v1/workflows/migratable/{appID}/]
4. The fourth and final endpoint allows you to toggle the "ready to migrate" flag on a given workflow back to false in the source environment after the workflow has been successfuly published in the target environment.

The flow of these migration will typically follow the below:

Get list of apps ready to migrate ->
Loop through list and for each app returned ->
    Download individual workflow from source environment ->
    Publish workflow in target environment ->
    Reset "ready to migrate" flag on workflow in source environment

This functionality was built into API endpoints in order to give the customer ultimate flexibility in designing their own migration process. An example C# script is attached to this post to help demonstrate how a simple migration could occur. This code is just a sample however, and should not be used for your own migration process.

Attachments
Comments
Alteryx Certified Partner
Alteryx Certified Partner

This is fantastic! I have a few questions.

 

First, I've upgraded to 2018.4 but I do not see these endpoints in the interactive documentation: https://gallery.alteryx.com/api-docs/

Are there plans to add them?

 

Second, even though this was intended to migrate between environments, could it also be used to migrate workflows within environments? For example, if I have a workflow on Gallery and I want to switch the workflow owner from one private studio to another, could I do that with the workflow POST endpoint? I noticed that the form has an "owner" section.

Alteryx
Alteryx

Hi @michael_treadwell,

 

Thanks for the feedback, below I'll try to answer both of the questions that you have:

 

1) We were mindful of not putting the new endpoints on this page for right now, since they don't have to do with a single server install and more to do with multiple installs and we didn't want to confuse anyone with putting them in the same place. That being said, we might add them somewhere else on that page going forward.

 

2) Yes, it's entirely possible to use the new endpoints with your server but it would essentially "copy" the workflow over to a new studio. Note that any schedules that were pointed to the old workflow will not know anything about the new workflow.

 

Let me know if you have any further questions.

Thanks!

Jeff

Hi @JeffreyP,

 

We have had a controller/scheduler component (version 2018.4) running on a Windows 7 workstation, while we were piloting the product.

 

We recently purchased an Alteryx Server and are in the midst of setting it up. Once the Alteryx Server is up and running, how do I transfer all the workflows that are stored and scheduled in the controller into the Alteryx Server? 

 

Is there a quick migration tool I could use?

 

Thanks!

Alteryx
Alteryx

Hi @sameerpremji,

 

Thanks for reaching out to me. As long as the schedules from Alteryx Designer were setup to run in the Scheduler DB (see image below), there is a way to migrate them to over to Alteryx Server...

schedule.png

You will either need to have admin access to your server or have someone else set this up, but here are the steps. Click on the Jobs navigation tab on the left, then click on the Migrate tab (as long as schedules are enabled on server), you should see these schedules in the list and you can migrate them onto your new server. Once you migrate them, there is no undoing, so make sure you select the correct person to own the schedules.

 

schedule2.png

 

Thanks!
Jeff

 

Hi @JeffreyP,

 

Correct me if I'm wrong but based on your above response, I get the impression that you're suggesting a way of migrating workflows from one server to another. This is not that case in my situation.

 

There is no "Gallery Admin Portal" on the Windows 7 workstation, so I can't browse and migrate the scheduled jobs/workflows by navigating to 'Jobs' section on a browser. That workstation only contained Alteryx Designer with Scheduler install. So, how can I migrate the jobs from the workstation (without any server component) to an Alteryx Server running a Web portal?

 

Also, on another note, I couldn't find anywhere in setup/install/configure documentation as to whether to first install a Web Server (IIS or Apache) on an Alteryx Server first or does the Alteryx Server setup utility installs the Web Server for you automatically or not?

 

 

Alteryx
Alteryx

Hi @sameerpremji,

 

It sounds like you have a workstation with Designer and schedules on it , you're setting up an Alteryx Server right now, and you want to migrate those old schedules over to the new Alteryx Server? We should be able to help you get these schedules migrated as long as they were setup with that first option to run in the DB and not on disk. I suggest you reach out to Alteryx Support at support@alteryx.com. They should be able to answer all of your questions there and walk you through this process more clearly.

 

Also, to answer your other question, Alteryx Server does not run on Apache or IIS, it starts itself up. 

 

Thanks,
Jeff

Hi @JeffreyP,

 

That is correct -- yes, all of our schedules were setup with that first option to run in the DB and not on disk. I'll contact Support as you've suggested.

 

So, the Alteryx Server software automatically opens up Ports 80 and 443 and starts serving web pages in the Admin Web Portal?

 

Do you have a link to the instructions on how to setup SSL Certificate so that we can hit the Admin Portal via HTTPS (Port 443)?

 

Thanks a lot :)

Alteryx
Alteryx

Hi @sameerpremji,

 

Yes, once the server configuration process is complete it should automatically start up a new service called AlteryxService, that will be hosted on port 40 (by default).

 

Here is an article that one of our customer support team members wrote about SSL:

https://community.alteryx.com/t5/Alteryx-Server-Knowledge-Base/Configuring-Alteryx-Server-for-SSL-Ob...

 

Feel free to bring up any other questions you may have with our support team.

 

Thanks,

Jeff

Thanks a lot @JeffreyP for your help.

Asteroid

@JeffreyP,

 

This is pretty cool. I have a ways before I can actual try this out, but I'm assuming there is a way just like you set the workertag and canDownload, to set the run modes.

I'm thinking of changing the default to safe mode on our Private gallery and only ones that we have migrated over are set to unrestricted mode.

 

Any thoughts or suggestions? Good idea, bad idea?

set.jpg

Alteryx
Alteryx

Hi @levell_x_dunn,

 

We actually do not have a run mode option when migrating workflows, this is definitely something we can look into for a future update though. Nice suggestion! Currently I believe all the workflows are migrated over with the run mode that is set in the Server configurations. If you wanted to do this right now, there would be a lot of manual work involved; every time you migrate a workflow over to the new Server you would have to change the run mode settings on that workflow.

 

Thanks!
Jeff

Meteoroid

@JeffreyP,

 

I have a question regarding the last step of the solution about migrating worflows from one environment (Let's say sandbox) to production (let's say Live).

 

when the process calls the API to PUT (publish the workflow in the new environment), the process actually copy the workflow on the new enviorement, what happens if we have an old version of that workflow on the destination enviroment? we would have a second copy or a new version will populates on the destination?

 

Also, another question about the functionality of Migrate on: Gallery Admin>Jobs>Migrate after Schedule Workflow on my computer checking the option "Run a copy of the worflow stored in the Scheduler DB". Is this option another way to do the migration? which is the best option? the API option (that you explain at the beginning of this post) or this second option?

 

Thanks a lot for your time,

Alteryx
Alteryx

Hi @jsoler,

 

Thanks for reaching out to me. You are correct, by default this will create a new workflow in your destination environment, so if you already had one, there will now be two of them now. However, with this API you can provide a 'sourceId' (the workflow id in the destination environment) and then it will overwrite the existing workflow with the new one that you copied over.

 

Similarly, yes. You could use the Scheduler option in Designer and point to the Live environment to copy it over. The thought here is that you'll have to manually do each workflow one at a time via Designer. And with the API it would be possible to write code such that it would copy as many workflows as you wanted to.

 

Thanks,

Jeff

Hi @JeffreyP ,

 

We have an Alteryx Gallery Server on-premise and we have a few folks with Alteryx Designer who publish workflows onto the Gallery. 

 

An Analyst typically creates and designs a workflow and saves this file locally on their PC. Once they're happy, the publish it to the on-premise Gallery. However, if there are some enhancement to be made, the Analyst opens the same locally stored file and makes the necessary changes.

 

When they publish the same workflow file onto the Gallery, it does not prompt us that "A workflow with this name already exists. Do you want to overwrite?". Instead, it actually publishes the enhanced workflow file, although with the same name, as a new workflow. This results in us cleaning up duplicates as well as re-configuring specific permissions for collections, users, etc.

 

The GUI method does not allow us to "overwrite" the same workflow file, which I think only encourages duplicates and extra-work. We also use Tableau Server and when Tableau Publishers use Tableau Desktop to publish their workbooks, the server actually detects the same workbook name and asks/prompts the publisher/author whether to overwrite or create a totally new workbook.

 

I have not had a need to use the API method as we mainly use the Designer tool to publish our workflows onto our Gallery.

 

Any advise? We also have created this enhancement request as an idea and we're hoping someone will notice it.

Alteryx
Alteryx

Hi @sameerpremji ,

 

My recommendation would be to actually have your users open the workflow in Designer from the Gallery, rather than from their local computer. The Gallery has no knowledge of local files on your computer. And you are correct, file names are not unique on the Gallery, this was done by design.

 

The steps I would take here; open up Designer, go to File -> Open Workflow -> Select your Gallery -> Select the workflow. Then after you make your changes/edits to the workflow, go to File -> Save, and this will treat it as a new version of that workflow, instead of treating it as a Save As for a new workflow. Essentially treating the Gallery as your file repository instead of your local computers, since you also have version control in the Gallery, you can set which version is "published" (ie. which version your viewers will be executing on the Gallery).

 

Thanks,
Jeff

Alteryx
Alteryx

@JeffreyP Any recommendations on which code compiler to use?

Alteryx
Alteryx

Hi @AmeliaG ,

 

It's really to the users discretion. As long as it supports multi-part forms. But I would recommend using whatever they're comfortable with.

 

Thanks,

Jeff

Alteryx
Alteryx

Hi @AmeliaG 

Here is an export version as an Alteryx Macro.  Yes it can be done without R or Python with just the native tools within designer. However, if you are like me with some self-signed certs you are probably going to run into a problem.  So, this is a quick python based macro that would allow you to override the SSL verification process.  

 

config.png

 

workflow.png

 

results.png

 

 

 

 

#################################
from ayx import Package
from ayx import Alteryx


#################################
import requests, urllib, hmac, hashlib
import collections, base64, math, random, string, time, sys, os
import pandas as pd


#################################
params = Alteryx.read('params')
params = params.iloc[0]
oauth_consumer_key = params['oauth_consumer_key']
oauth_consumer_secret = params['oauth_consumer_secret']
gallery_migrate_out = params['gallery_migrate_out']
request_proto = 'https://' if params['gallery_migrate_out_SSL'] > 0 else 'http://'
ssl_verify = False if params['gallery_migrate_out_self_signed_SSL'] > 0 else True
ayx_stage_directory = os.path.join(os.path.normpath(params['workflow_stage_directory']), '')



#################################
class Gallery(object):
    def __init__(self, apiLocation, apiKey, apiSecret):
        self.apiLocation = request_proto + apiLocation + '/gallery/api/admin/v1'
        self.apiKey = apiKey
        self.apiSecret = apiSecret

        
    """Generate pseudo-random number"""

    def generate_nonce(self, length=11):
        return ''.join([str(random.choice(string.ascii_uppercase + string.digits + string.ascii_lowercase)) for i in
                        range(length)])

    
    """Returns HMAC-SHA1 signature"""

    def generateSignature(self, httpMethod, url, params):
        q = lambda x: requests.utils.quote(x, safe="~")
        sorted_params = collections.OrderedDict(sorted(params.items()))

        normalized_params = urllib.parse.urlencode(sorted_params)
        base_string = "&".join((httpMethod.upper(), q(url), q(normalized_params)))

        # Python 3 requires string in bytes for hmac.new()
        secret_bytes = bytes("&".join([self.apiSecret, '']), 'ascii')
        base_bytes = bytes(base_string, 'ascii')
        sig = hmac.new(secret_bytes, base_bytes, hashlib.sha1)

        # Python 3 requires use of b64encode method from base64
        return base64.b64encode(sig.digest())
 

    """Constructs OAuth Parameters"""

    def buildOAuthParams(self):
        return {'oauth_consumer_key': self.apiKey,
                'oauth_signature_method': 'HMAC-SHA1',
                'oauth_timestamp': str(int(math.floor(time.time()))),
                'oauth_nonce': self.generate_nonce(11),
                'oauth_version': '1.0'}

    
    """Finds workflows that have been flagged for migration"""
    
    def migrate (self):
        method = 'GET'
        url = self.apiLocation + '/workflows/migratable/'
        params = self.buildOAuthParams()
        signature = self.generateSignature(method, url, params)
        params.update({'oauth_signature': signature})

        try:
            output = requests.get(url, params=params, verify=ssl_verify)
            output.raise_for_status()
        except requests.exceptions.HTTPError as err:
            print(err)
            sys.exit(err)
        except requests.exceptions.RequestException as err2:
            print(err2)
            sys.exit(err2)

        return output, output.json()
    
    
    """Exports the App that was requested"""

    def getApp(self, appId):
        method = 'GET'
        url = self.apiLocation + '/' + appId + '/package/'
        params = self.buildOAuthParams()
        signature = self.generateSignature(method, url, params)
        params.update({'oauth_signature': signature})

        try:
            local_filename = ayx_stage_directory + appId + '.yxzp'
            # NOTE the stream=True parameter below
            with requests.get(url, stream=True, params=params, verify=ssl_verify) as yxzp:
                yxzp.raise_for_status()
                with open(local_filename, 'wb') as f:
                    for chunk in yxzp.iter_content(chunk_size=8192):
                        if chunk: # filter out keep-alive new chunks
                            f.write(chunk)
                            # f.flush() 
                            
        except requests.exceptions.HTTPError as err:
            print(err)
            sys.exit(err)
        except requests.exceptions.RequestException as err2:
            print(err2)
            sys.exit(err2)
        return
            

    
    


#################################
con = Gallery(gallery_migrate_out, oauth_consumer_key, oauth_consumer_secret)
response, data = con.migrate()
migratableWorkflows = response.json()
df = pd.DataFrame (migratableWorkflows)
df.rename(columns={'id':'appId'}, inplace=True)
df['fullPath'] = ayx_stage_directory.replace("/","\\") + df['appId']  +'.yxzp'
for row in df.itertuples():
    pkg = con.getApp(row.appId)
        


#################################
Alteryx.write(df,1)

 

Atom

@PeterA is it possible to download this macro from somewhere?  I set up a Sandbox and Production environment at my company, but have been struggling to figure out how to enable migrating workflows from one environment to another.

Asteroid

I agree. @PeterA could you please if possible make this macro available on the community site? Also does this just migrate apps (.yxwz) files? Hopefully any resources available on gallery like yxmd, yxmc or yxwz files can be downloaded and pushed to target environment.

Aurora
Aurora

If anyone is struggling to get the workflow uploader up and running in Postman - here are the steps to get this working:

 

Step 1: Get a simple call working with oAuth

First get connected to your server using OAuth authentication, with the appropriate keys - and confirm that you're getting a good connection using oAuth Authentication on the GET method for workflows.

- Add in the API URL for workflow list

  • you can find this on <your gallery>\api-docs.   for example: http:\\localhost\gallery\api-docs)
  • On my local server, the workflow endpoint is http://localhost/gallery/api/admin/v1/workflows/

Set the Authorization on the auth tab

  • use oAuth 1
  • You need the admin key and the admin secret (blurred out below) which you get from the gallery admin
  • all the rest leave as default

 

Postman - 1 - good call to server.png

 

 

You can tell you're getting a good response because of the highlighted "200" server status; and the response in the body which shows some of your workflows.

 

Step 2: Change this to an upload call

  • Set up the POST query using exactly the same OAuth credentials and URL for the API
  • Use the exact same API endpoint - \workflows.    For example: http://localhost/gallery/api/admin/v1/workflows/
  • Then change the body to "form-data"
  • Set the first key to be "file" and hover over the end to change type from "text" to "file" and select your packaged file to upload
  • Add in the remaining form components per the screenshot or per @JeffreyP 's C# code attached above
  • Then you should get a good response from the server - it will return your new AppID

Postman-2-PostCallSucceeds.png

 

Open Question: We're still experimenting with the SourceID to see if we can get it to override the canvas instead of duplicating a new one - but no-luck so-far.

 

cc: @adams_ca ; @revathi ; @Kosi @Sotoll ; @PeterA 

Meteor

Thank @SeanAdams !


I have found that if you specify the sourceId for a workflow that already exists, even though the expected behavior is that the workflow will be updated/overwritten, it is in fact not.  A new workflow is always created using this API.

 

Any ideas/solutions on how to ensure that a new workflow isn't published to the gallery when that workflow already exists?

 

Alteryx
Alteryx

Hi @adams_ca,

 

I suspect something may be off with the body in your gallery POST call - as it should overwrite the workflow.  I've found that spaces in the appID and/or not getting the case correct for my keys (e.g. must be "sourceId" not "SourceID) will however generate a new workflow.   

 

Now this may be a good thing or a bad thing.  What happens is when you do get the sourceId correct it will overwrite the workflow, and unfortunately setting latest version is 1 & published version is 1 - with no previous history is preserved.   @JeffreyP, is there a method to get versioning to work?

Alteryx
Alteryx

Hi @PeterA ,

 

When you migrate a workflow from one server to another and you supply a sourceId, it's assumed that you want to replace that workflow, not add a new version of that workflow. It's the same as the replace functionality that's currently on the Gallery. This was by design. So currently there is no way to add versions of the same workflow. The idea behind it was that you'd already do this in the target environment and when you promote the workflow to the source environment, you'd already know the version that you were promoting and wouldn't need to keep track of it there. But we're always open to discussions if you think this is something that is widely wanted.

 

Thanks,
Jeff

Aurora
Aurora

Thank you @JeffreyP  - this is definitely necessary - even from a regulatory and roll-back perspective, we have to be able to keep versions.    The regulatory angle is that under SOX we need to keep an active audit trail of changes to prod assets.    The roll-back perspective is that you need the ability to roll-back to a previously known good version.

 

So yes - the best behavior would be to reversion the asset with the provided App ID so that each version is kept along with an audit trail.