Alteryx Server Knowledge Base

Definitive answers from Server experts.

Migrating Workflows

Jefypop
Alteryx
Alteryx
Created

A frequent feature request we hear from customers is the ability to do true siloed environments when developing and testing workflows. Alteryx Server already has a versioning feature which allows you to publish a specific revision of a workflow, however this can lead to confusion and potentially incorrect versions being run when a large number of workflows and users exist in a system.

To make versioning of workflows easier we have now included functionality that allows for migrating workflows between multiple Alteryx Server environments.

In Alteryx Server 2018.4 you can now manage separate environments and deploy a workflow from one environment to the next. This allows for completely siloed environments, and gives customers greater ability to manage the lifecyle of their workflows.

How does it work?

Within the admin portal of Alteryx Server, there is now a new option under the Workflow section to enable Workflow Migrations. This global flag will enable a set of controls at the workflow level to mark a workflow as "Ready for migration". Workflows can be marked as ready for migration in one of two places.

1. In the admin portal, under the Workflows section. Within a workflow, you will see an option to mark the workflow as "This workflow is ready to be migrated".
2. Within your private Studio under the "Workflow Settings" dialog for a specific workflow.

The core functionality of the migration is handled in four new endpoints. Three of the endpoints will be in your source environment (where workflows will be migrated from) and one of the endpoints exists in the target environment (where you will be publishing workflows to). All of these endpoints authenticate using our Oauth1 auth scheme just like all of our other public admin API endpoints.

These four endpoints are:

[Source Environment]
[HTTP GET]
[api/admin/v1/workflows/migratable/?subscriptionIds={comma separated subscriptionIds}/]
1. The first endpoint exposes an array of ApplicationIds which have been marked as ready to migrate. This endpoint takes a comma separated list of subscriptionIds as a query parameter and will return all workflows marked as ready to migrate under the specific studio(s). If no subscriptionsIds are provided then all workflows that have been marked as ready to migrate should be returned. This is a lightweight endpoint that will return just three properties. The appId, the currently published revision ID, as well as the subscriptionID the workflow belongs to.

[Source Environment]
[HTTP GET]
[api/admin/v1/{appID}/package/]
2. The second endpoint will download the actual workflow as a .YXZP. This endpoint takes an appID as a parameter and downloads the entire workflow as a package.

[Target Environment]
[HTTP POST]
[api/admin/v1/workflows/]
3. The third endpoint is a publish endpoint. This allows you to take the workflow obtained in the second endpoint and publish to the target environment. This post uses a multipart form-data request. The form sections and properties can be found in the code example.

[Source Environment]
[HTTP PUT]
[api/admin/v1/workflows/migratable/{appID}/]
4. The fourth and final endpoint allows you to toggle the "ready to migrate" flag on a given workflow back to false in the source environment after the workflow has been successfuly published in the target environment.

The flow of these migration will typically follow the below:

Get list of apps ready to migrate ->
Loop through list and for each app returned ->
Download individual workflow from source environment ->
Publish workflow in target environment ->
Reset "ready to migrate" flag on workflow in source environment

This functionality was built into API endpoints in order to give the customer ultimate flexibility in designing their own migration process. An example C# script is attached to this post to help demonstrate how a simple migration could occur. This code is just a sample however, and should not be used for your own migration process.

***Attached is also a proof of concept of using these new endpoints. It is not supported, but just an example of how to use the new APIs! You will need to enter the target and source consumer keys and secrets (marked by red annotations) in order for this to work. And it will only migrate workflows that have been marked ready for migration***

Attachments
Comments
michael_treadwell
12 - Quasar
12 - Quasar

This is fantastic! I have a few questions.

 

First, I've upgraded to 2018.4 but I do not see these endpoints in the interactive documentation: https://gallery.alteryx.com/api-docs/

Are there plans to add them?

 

Second, even though this was intended to migrate between environments, could it also be used to migrate workflows within environments? For example, if I have a workflow on Gallery and I want to switch the workflow owner from one private studio to another, could I do that with the workflow POST endpoint? I noticed that the form has an "owner" section.

Jefypop
Alteryx
Alteryx

Hi @michael_treadwell,

 

Thanks for the feedback, below I'll try to answer both of the questions that you have:

 

1) We were mindful of not putting the new endpoints on this page for right now, since they don't have to do with a single server install and more to do with multiple installs and we didn't want to confuse anyone with putting them in the same place. That being said, we might add them somewhere else on that page going forward.

 

2) Yes, it's entirely possible to use the new endpoints with your server but it would essentially "copy" the workflow over to a new studio. Note that any schedules that were pointed to the old workflow will not know anything about the new workflow.

 

Let me know if you have any further questions.

Thanks!

Jeff

sameerpremji
8 - Asteroid

Hi @Jefypop,

 

We have had a controller/scheduler component (version 2018.4) running on a Windows 7 workstation, while we were piloting the product.

 

We recently purchased an Alteryx Server and are in the midst of setting it up. Once the Alteryx Server is up and running, how do I transfer all the workflows that are stored and scheduled in the controller into the Alteryx Server? 

 

Is there a quick migration tool I could use?

 

Thanks!

Jefypop
Alteryx
Alteryx

Hi @sameerpremji,

 

Thanks for reaching out to me. As long as the schedules from Alteryx Designer were setup to run in the Scheduler DB (see image below), there is a way to migrate them to over to Alteryx Server...

schedule.png

You will either need to have admin access to your server or have someone else set this up, but here are the steps. Click on the Jobs navigation tab on the left, then click on the Migrate tab (as long as schedules are enabled on server), you should see these schedules in the list and you can migrate them onto your new server. Once you migrate them, there is no undoing, so make sure you select the correct person to own the schedules.

 

schedule2.png

 

Thanks!
Jeff

 

sameerpremji
8 - Asteroid

Hi @Jefypop,

 

Correct me if I'm wrong but based on your above response, I get the impression that you're suggesting a way of migrating workflows from one server to another. This is not that case in my situation.

 

There is no "Gallery Admin Portal" on the Windows 7 workstation, so I can't browse and migrate the scheduled jobs/workflows by navigating to 'Jobs' section on a browser. That workstation only contained Alteryx Designer with Scheduler install. So, how can I migrate the jobs from the workstation (without any server component) to an Alteryx Server running a Web portal?

 

Also, on another note, I couldn't find anywhere in setup/install/configure documentation as to whether to first install a Web Server (IIS or Apache) on an Alteryx Server first or does the Alteryx Server setup utility installs the Web Server for you automatically or not?

 

 

Jefypop
Alteryx
Alteryx

Hi @sameerpremji,

 

It sounds like you have a workstation with Designer and schedules on it , you're setting up an Alteryx Server right now, and you want to migrate those old schedules over to the new Alteryx Server? We should be able to help you get these schedules migrated as long as they were setup with that first option to run in the DB and not on disk. I suggest you reach out to Alteryx Support at support@alteryx.com. They should be able to answer all of your questions there and walk you through this process more clearly.

 

Also, to answer your other question, Alteryx Server does not run on Apache or IIS, it starts itself up. 

 

Thanks,
Jeff

sameerpremji
8 - Asteroid

Hi @Jefypop,

 

That is correct -- yes, all of our schedules were setup with that first option to run in the DB and not on disk. I'll contact Support as you've suggested.

 

So, the Alteryx Server software automatically opens up Ports 80 and 443 and starts serving web pages in the Admin Web Portal?

 

Do you have a link to the instructions on how to setup SSL Certificate so that we can hit the Admin Portal via HTTPS (Port 443)?

 

Thanks a lot 🙂

Jefypop
Alteryx
Alteryx

Hi @sameerpremji,

 

Yes, once the server configuration process is complete it should automatically start up a new service called AlteryxService, that will be hosted on port 40 (by default).

 

Here is an article that one of our customer support team members wrote about SSL:

https://community.alteryx.com/t5/Alteryx-Server-Knowledge-Base/Configuring-Alteryx-Server-for-SSL-Ob...

 

Feel free to bring up any other questions you may have with our support team.

 

Thanks,

Jeff

sameerpremji
8 - Asteroid

Thanks a lot @Jefypop for your help.

levell_x_dunn
9 - Comet

@Jefypop,

 

This is pretty cool. I have a ways before I can actual try this out, but I'm assuming there is a way just like you set the workertag and canDownload, to set the run modes.

I'm thinking of changing the default to safe mode on our Private gallery and only ones that we have migrated over are set to unrestricted mode.

 

Any thoughts or suggestions? Good idea, bad idea?

set.jpg

Jefypop
Alteryx
Alteryx

Hi @levell_x_dunn,

 

We actually do not have a run mode option when migrating workflows, this is definitely something we can look into for a future update though. Nice suggestion! Currently I believe all the workflows are migrated over with the run mode that is set in the Server configurations. If you wanted to do this right now, there would be a lot of manual work involved; every time you migrate a workflow over to the new Server you would have to change the run mode settings on that workflow.

 

Thanks!
Jeff

jsoler
7 - Meteor

@JeffreyP,

 

I have a question regarding the last step of the solution about migrating worflows from one environment (Let's say sandbox) to production (let's say Live).

 

when the process calls the API to PUT (publish the workflow in the new environment), the process actually copy the workflow on the new enviorement, what happens if we have an old version of that workflow on the destination enviroment? we would have a second copy or a new version will populates on the destination?

 

Also, another question about the functionality of Migrate on: Gallery Admin>Jobs>Migrate after Schedule Workflow on my computer checking the option "Run a copy of the worflow stored in the Scheduler DB". Is this option another way to do the migration? which is the best option? the API option (that you explain at the beginning of this post) or this second option?

 

Thanks a lot for your time,

Jefypop
Alteryx
Alteryx

Hi @jsoler,

 

Thanks for reaching out to me. You are correct, by default this will create a new workflow in your destination environment, so if you already had one, there will now be two of them now. However, with this API you can provide a 'sourceId' (the workflow id in the destination environment) and then it will overwrite the existing workflow with the new one that you copied over.

 

Similarly, yes. You could use the Scheduler option in Designer and point to the Live environment to copy it over. The thought here is that you'll have to manually do each workflow one at a time via Designer. And with the API it would be possible to write code such that it would copy as many workflows as you wanted to.

 

Thanks,

Jeff

sameerpremji
8 - Asteroid

Hi @Jefypop ,

 

We have an Alteryx Gallery Server on-premise and we have a few folks with Alteryx Designer who publish workflows onto the Gallery. 

 

An Analyst typically creates and designs a workflow and saves this file locally on their PC. Once they're happy, the publish it to the on-premise Gallery. However, if there are some enhancement to be made, the Analyst opens the same locally stored file and makes the necessary changes.

 

When they publish the same workflow file onto the Gallery, it does not prompt us that "A workflow with this name already exists. Do you want to overwrite?". Instead, it actually publishes the enhanced workflow file, although with the same name, as a new workflow. This results in us cleaning up duplicates as well as re-configuring specific permissions for collections, users, etc.

 

The GUI method does not allow us to "overwrite" the same workflow file, which I think only encourages duplicates and extra-work. We also use Tableau Server and when Tableau Publishers use Tableau Desktop to publish their workbooks, the server actually detects the same workbook name and asks/prompts the publisher/author whether to overwrite or create a totally new workbook.

 

I have not had a need to use the API method as we mainly use the Designer tool to publish our workflows onto our Gallery.

 

Any advise? We also have created this enhancement request as an idea and we're hoping someone will notice it.

Jefypop
Alteryx
Alteryx

Hi @sameerpremji ,

 

My recommendation would be to actually have your users open the workflow in Designer from the Gallery, rather than from their local computer. The Gallery has no knowledge of local files on your computer. And you are correct, file names are not unique on the Gallery, this was done by design.

 

The steps I would take here; open up Designer, go to File -> Open Workflow -> Select your Gallery -> Select the workflow. Then after you make your changes/edits to the workflow, go to File -> Save, and this will treat it as a new version of that workflow, instead of treating it as a Save As for a new workflow. Essentially treating the Gallery as your file repository instead of your local computers, since you also have version control in the Gallery, you can set which version is "published" (ie. which version your viewers will be executing on the Gallery).

 

Thanks,
Jeff

AmeliaG
Alteryx
Alteryx

@Jefypop Any recommendations on which code compiler to use?

Jefypop
Alteryx
Alteryx

Hi @AmeliaG ,

 

It's really to the users discretion. As long as it supports multi-part forms. But I would recommend using whatever they're comfortable with.

 

Thanks,

Jeff

PeterA
Alteryx
Alteryx

Hi @AmeliaG 

Here is an export version as an Alteryx Macro.  Yes it can be done without R or Python with just the native tools within designer. However, if you are like me with some self-signed certs you are probably going to run into a problem.  So, this is a quick python based macro that would allow you to override the SSL verification process.  

 

config.png

 

workflow.png

 

results.png

 

 

 

 

#################################
from ayx import Package
from ayx import Alteryx


#################################
import requests, urllib, hmac, hashlib
import collections, base64, math, random, string, time, sys, os
import pandas as pd


#################################
params = Alteryx.read('params')
params = params.iloc[0]
oauth_consumer_key = params['oauth_consumer_key']
oauth_consumer_secret = params['oauth_consumer_secret']
gallery_migrate_out = params['gallery_migrate_out']
request_proto = 'https://' if params['gallery_migrate_out_SSL'] > 0 else 'http://'
ssl_verify = False if params['gallery_migrate_out_self_signed_SSL'] > 0 else True
ayx_stage_directory = os.path.join(os.path.normpath(params['workflow_stage_directory']), '')



#################################
class Gallery(object):
    def __init__(self, apiLocation, apiKey, apiSecret):
        self.apiLocation = request_proto + apiLocation + '/gallery/api/admin/v1'
        self.apiKey = apiKey
        self.apiSecret = apiSecret

        
    """Generate pseudo-random number"""

    def generate_nonce(self, length=11):
        return ''.join([str(random.choice(string.ascii_uppercase + string.digits + string.ascii_lowercase)) for i in
                        range(length)])

    
    """Returns HMAC-SHA1 signature"""

    def generateSignature(self, httpMethod, url, params):
        q = lambda x: requests.utils.quote(x, safe="~")
        sorted_params = collections.OrderedDict(sorted(params.items()))

        normalized_params = urllib.parse.urlencode(sorted_params)
        base_string = "&".join((httpMethod.upper(), q(url), q(normalized_params)))

        # Python 3 requires string in bytes for hmac.new()
        secret_bytes = bytes("&".join([self.apiSecret, '']), 'ascii')
        base_bytes = bytes(base_string, 'ascii')
        sig = hmac.new(secret_bytes, base_bytes, hashlib.sha1)

        # Python 3 requires use of b64encode method from base64
        return base64.b64encode(sig.digest())
 

    """Constructs OAuth Parameters"""

    def buildOAuthParams(self):
        return {'oauth_consumer_key': self.apiKey,
                'oauth_signature_method': 'HMAC-SHA1',
                'oauth_timestamp': str(int(math.floor(time.time()))),
                'oauth_nonce': self.generate_nonce(11),
                'oauth_version': '1.0'}

    
    """Finds workflows that have been flagged for migration"""
    
    def migrate (self):
        method = 'GET'
        url = self.apiLocation + '/workflows/migratable/'
        params = self.buildOAuthParams()
        signature = self.generateSignature(method, url, params)
        params.update({'oauth_signature': signature})

        try:
            output = requests.get(url, params=params, verify=ssl_verify)
            output.raise_for_status()
        except requests.exceptions.HTTPError as err:
            print(err)
            sys.exit(err)
        except requests.exceptions.RequestException as err2:
            print(err2)
            sys.exit(err2)

        return output, output.json()
    
    
    """Exports the App that was requested"""

    def getApp(self, appId):
        method = 'GET'
        url = self.apiLocation + '/' + appId + '/package/'
        params = self.buildOAuthParams()
        signature = self.generateSignature(method, url, params)
        params.update({'oauth_signature': signature})

        try:
            local_filename = ayx_stage_directory + appId + '.yxzp'
            # NOTE the stream=True parameter below
            with requests.get(url, stream=True, params=params, verify=ssl_verify) as yxzp:
                yxzp.raise_for_status()
                with open(local_filename, 'wb') as f:
                    for chunk in yxzp.iter_content(chunk_size=8192):
                        if chunk: # filter out keep-alive new chunks
                            f.write(chunk)
                            # f.flush() 
                            
        except requests.exceptions.HTTPError as err:
            print(err)
            sys.exit(err)
        except requests.exceptions.RequestException as err2:
            print(err2)
            sys.exit(err2)
        return
            

    
    


#################################
con = Gallery(gallery_migrate_out, oauth_consumer_key, oauth_consumer_secret)
response, data = con.migrate()
migratableWorkflows = response.json()
df = pd.DataFrame (migratableWorkflows)
df.rename(columns={'id':'appId'}, inplace=True)
df['fullPath'] = ayx_stage_directory.replace("/","\\") + df['appId']  +'.yxzp'
for row in df.itertuples():
    pkg = con.getApp(row.appId)
        


#################################
Alteryx.write(df,1)

 

jdm5650
5 - Atom

@PeterA is it possible to download this macro from somewhere?  I set up a Sandbox and Production environment at my company, but have been struggling to figure out how to enable migrating workflows from one environment to another.

neeleshapatil
8 - Asteroid

I agree. @PeterA could you please if possible make this macro available on the community site? Also does this just migrate apps (.yxwz) files? Hopefully any resources available on gallery like yxmd, yxmc or yxwz files can be downloaded and pushed to target environment.

SeanAdams
17 - Castor
17 - Castor

If anyone is struggling to get the workflow uploader up and running in Postman - here are the steps to get this working:

 

Step 1: Get a simple call working with oAuth

First get connected to your server using OAuth authentication, with the appropriate keys - and confirm that you're getting a good connection using oAuth Authentication on the GET method for workflows.

- Add in the API URL for workflow list

  • you can find this on <your gallery>\api-docs.   for example: http:\\localhost\gallery\api-docs)
  • On my local server, the workflow endpoint is http://localhost/gallery/api/admin/v1/workflows/

Set the Authorization on the auth tab

  • use oAuth 1
  • You need the admin key and the admin secret (blurred out below) which you get from the gallery admin
  • all the rest leave as default

 

Postman - 1 - good call to server.png

 

 

You can tell you're getting a good response because of the highlighted "200" server status; and the response in the body which shows some of your workflows.

 

Step 2: Change this to an upload call

  • Set up the POST query using exactly the same OAuth credentials and URL for the API
  • Use the exact same API endpoint - \workflows.    For example: http://localhost/gallery/api/admin/v1/workflows/
  • Then change the body to "form-data"
  • Set the first key to be "file" and hover over the end to change type from "text" to "file" and select your packaged file to upload
  • Add in the remaining form components per the screenshot or per @Jefypop 's C# code attached above
  • Then you should get a good response from the server - it will return your new AppID

Postman-2-PostCallSucceeds.png

 

Open Question: We're still experimenting with the SourceID to see if we can get it to override the canvas instead of duplicating a new one - but no-luck so-far.

 

cc: @adams_ca ; @revathi ; @Kosi @Sotoll ; @PeterA 

adams_ca
8 - Asteroid

Thank @SeanAdams !


I have found that if you specify the sourceId for a workflow that already exists, even though the expected behavior is that the workflow will be updated/overwritten, it is in fact not.  A new workflow is always created using this API.

 

Any ideas/solutions on how to ensure that a new workflow isn't published to the gallery when that workflow already exists?

 

PeterA
Alteryx
Alteryx

Hi @adams_ca,

 

I suspect something may be off with the body in your gallery POST call - as it should overwrite the workflow.  I've found that spaces in the appID and/or not getting the case correct for my keys (e.g. must be "sourceId" not "SourceID) will however generate a new workflow.   

 

Now this may be a good thing or a bad thing.  What happens is when you do get the sourceId correct it will overwrite the workflow, and unfortunately setting latest version is 1 & published version is 1 - with no previous history is preserved.   @JeffreyP, is there a method to get versioning to work?

Jefypop
Alteryx
Alteryx

Hi @PeterA ,

 

When you migrate a workflow from one server to another and you supply a sourceId, it's assumed that you want to replace that workflow, not add a new version of that workflow. It's the same as the replace functionality that's currently on the Gallery. This was by design. So currently there is no way to add versions of the same workflow. The idea behind it was that you'd already do this in the target environment and when you promote the workflow to the source environment, you'd already know the version that you were promoting and wouldn't need to keep track of it there. But we're always open to discussions if you think this is something that is widely wanted.

 

Thanks,
Jeff

SeanAdams
17 - Castor
17 - Castor

Thank you @Jefypop  - this is definitely necessary - even from a regulatory and roll-back perspective, we have to be able to keep versions.    The regulatory angle is that under SOX we need to keep an active audit trail of changes to prod assets.    The roll-back perspective is that you need the ability to roll-back to a previously known good version.

 

So yes - the best behavior would be to reversion the asset with the provided App ID so that each version is kept along with an audit trail.

 

piyush
6 - Meteoroid
Did anyone implement the solution Jeffrey provide in their organisation?
jdm5650
5 - Atom

To anyone still looking for a solution around how to use these endpoints, Michael Treadwell (@michael_treadwell) had a session on this at Inspire 2019 and posted a link on the Interworks blog with macros that you can download and use to 1. pull all workflows that are ready to be migrated in a source environment and 2. publish those workflows to a target environment and reset the migration ready flags in the source environment.  After months of looking for an answer, I was finally about to set this up for my company's environment this morning because of Michael's session and resources.  Hopefully me sharing this here will help others.

 

https://interworks.com/blog/mtreadwell/2019/06/12/the-migratory-patterns-of-the-common-alteryx-workf...

 

(Personal note on implementation:  Make sure you pay attention to the part at the end where he says to make sure all the studios involved are paid type and have an expiration date.  This hung me up for an hour or so today.)

 

*Edited intial post to correctly tag Michael

neeleshapatil
8 - Asteroid

Hi 

While running Gallery Migration Workflow uploaded by JeffreyP received following error. Could you please confirm if there is any specific .exe file required to be placed and what would be it’s path?

 

Error: PublishWorkflow (18): Record #1: Tool #1: Failed to run external program "..\Supporting_Macros\AlteryxGalleryAPI_PublishWorkflow.exe": The system cannot find the file specified.

 

I also see as per below screenshot that command arguments has some hard coded values like gallery server etc. How it gets updated relevant to our sever details?

Image.jpg

 

Jefypop
Alteryx
Alteryx

Hi @neeleshapatil ,

 

If you downloaded the attached workflow, you will need to make sure that the executable file (AlteryxGalleryAPI_PublishWorkflow.exe) is a part of the package. Since the run command tool is inside the macro, these hard-coded values will be updated when it runs. The only things you should have to update are the server key/secret and the url for the source and target servers in the parent workflow.

 

Thanks!
Jeff

neeleshapatil
8 - Asteroid

Thanks JeffreyP for your quick reply.

But where i can find this executable file? Might me my company network preventing .exe file to be downloaded along with all other macros.

 

Also additional to this would like to know on the following point. 

 

There could be multiple private studio’s under source Alteryx server, we would like to get workflows from a particular private studio to be migrated to same private studio in the Target server. How this can be achieved?

As an Gallery admin, I can only see API key and secret of the private studio that I am part of on the source environment. How I can get API key and secret for other private studios so that we can migrate workflow from a particular private studio to same private studio in the Target server.

Jefypop
Alteryx
Alteryx

Yes, the executable file should be a part of the zip file that you downloaded. It's possible your company is preventing download of .exe's.

 

If this solution doesn't work for you it might be worth looking at the above post from Michael Treadwell's example.

 

To answer your other questions:

The key/secret should work server wide. They aren't specific to a users studios, rather they're for accessing the api's from an admin perspective. The actual tool you'll want to change is the formula tool - highlighted in yellow (it's currently hardcoded to my email address). Put the email you want to use for the user in there and it should work. Hope this makes sense!

Thanks again!
Jeff

GabsMuoio
5 - Atom

Hi Jeffrey,

 

First, congratulations by create this workflow, fantastic.

 

I have a problem when I migrated... The workflow is migrated, but not run, because show this error: Unable to translate alias ORACLE_UAT (I use data connections in gallery).

 

Can you help me?

 

Thanks

jason-sa
5 - Atom

Hello @Jefypop 

 

I am attempting to use the POST workflow API to move workflows from a dev environment to a prod environment. Everything is working great except the sourceId logic. Here is what I am experiencing.

 

  1. Develop new workflow in dev, and mark to migrate.
  2. Run the GET api to get the workflow appID
  3. Download the workflow with the GET api.
  4. POST the workflow with a blank sourceId to Prod which responds with the new appID of 1234
  5. Submit the PUT to reset the flag
  6. Make changes in Dev and then mark to migrate again
  7. Do steps 2-3
  8. POST the workflow with sourceId = 1234, but the response is appID of 5678. Subsequently, a new workflow is created even though the sourceId matches the appId in Prod.
  9. I re-submit the POST, with sourceId = 1234, and appID 5678 is now updated! 

I am confused by this behavior and what I am doing wrong. How is the server matching sourceId 1234 to appId 5678. I feel like I am using the wrong id in sourceId, but I don't know what would be the right ID. Any help is much appreciated.

camanoe
5 - Atom

Using this code, I had some issues and had to do some troubleshooting. 

The issue is that my server uses HTTPS.  I did a lot of comparing of this code to the code returned from the admin api section on the server.  I found when running the tests from the https://myserver.mycompany.com/gallery/api-docs/#!/workflows.json the request it generated added the :443, so I assumed that I needed to.

 

443.jpg

 

I found this post which ultimately solved my issue.  https://community.alteryx.com/t5/Dev-Space/Cannot-use-Gallery-API-Invalid-Signature/m-p/367731/highl...

 

A comment at the end by DanH pointed me in the right direction.

 

This was exactly my issue working withe the API.  We use SSL and I was using https://myserver.mycompany.com:443/gallery as my base connection.  I had to modify the macro and change it to https://myserver.mycompany.com/gallery  and after the encryption happened, I used a formula (Replace([Full_Request_URL], '.com', '.com:443'))to append the :443 to the url. 

fix.jpg

So to recap,the base URL should not contain the :443 but the URL should be modified after the SHA1 encryption has happened.

 

I struggled with this for days.

Matt

romacleod
5 - Atom

Hi Jeffrey,

 

Thanks for this post, we have found it very useful.  However, is there still an outstanding bug with overwriting workflows using sourceId?

 

Seems like many other users are reporting the same issue I have spotted (e.g. adams_ca) where setting the sourceId to an existing workflow id is causing a new workflow is created with a new id. 

 

If the POST is performed a second time using the same params, the newer workflow (width different id to sourceId) is updated and the original (with same id as sourceId) is unmodified. 

 

If I remove the first/original workflow from the gallery and then POST again using sourceId = old id, it still updates the newer/second copy that has a completely different id?!

 

This behaviour seems odd to me and trying to automate and manage duplicate workflows in the gallery with the same name is causing us quite a few headaches.  Is it actually possible to update an existing workflow in the gallery via the API without creating a new one first?  If this is a known bug is there any ETA when it might be resolved?

 

Many thanks,

 

Roger

rohitbandaru
5 - Atom

Hi Jeffrey,

       This post saves a lot of time. I have a similar kind of requirement for my organization in which I have to upload and be able to run the workflow through API. For that i have used your Program.cs then I am able to upload  a .yxzp file to alteryx gallery through api successfully. But the problem was when I am uploading the yxzp file directly to the gallery through UI, I was able to get the proper result from "/v1/workflows/{appId}/questions/" api call. But when I upload the same workflow through the api, the response for  "/v1/workflows/{appId}/questions/" is  "The file \"C:\\ProgramData\\Alteryx\\Service\\Staging\\XProcessCache\\XXXXXXXXXXXX\\Sample.yxmd\" does not exist.". Could you please help me to resolve this issue. Thanks In Advace. 

sricharan
7 - Meteor

I am working Alteryx API implementations using Python. i have executed all API endpoints(getting questions, executing the workflow, getting results of workflow, etc.) with the help of interactive api doc and Python.

 

I have stuck while Publishing workflow using API into another gallery(workflow migration). i am facing an issue - 

{"data":null,"exceptionName":"BadRequestException","innerExceptionMessage":"","message":"Must specify subscription key (oauth_consumer_key parameter)."}'.

 

Earlier while i was executing other endpoints Alteryx never asked me for subscription key.

For publishing, i am using,

  • .yxzp extended workflow as a example
  • URL - gallery/api/admin/v1/workflows/
  • admin API key & secret are used
  • passing file to upload in the files attribute in the requests.post method (e.g resp = requests.post(url, files ={'logo': file_to_publish}, headers = {
    'Content-Type': 'multipart/form-data','Accept': 'application/xml'}))
  • payload is file_to_publish

@JeffreyP and others Please help me to execute Publish endpoint? Could you please provide the artifacts(code/workflow) of the working prototype if you have?

 

Your reply is much appreciable.

KevinP
Alteryx
Alteryx

@GabsMuoio This is a known limitation. Gallery Data Connections are server specific and do not migrate with the workflows. Even if we did migrate them they would end up with a new ID number in the new environment and the workflow would still trigger this error. In order for these workflows to function they will need to be updated post migration to utilize an appropriate Gallery Data connection on the new Server.

 

@jason-sa and @romacleod If you haven't already please reach out to customer support to assist with investigating the issues you are encountering around the sourceid parameter. 

 

@rohitbandaru Can you also reach out to support for assistance investigating the behavior you are encountering.

 

@sricharan The error you are encountering is an authentication issue because the Oauth consumer key was either missing or incorrect. None of the endpoints to complete a migration should require the Studio/Subscription level key and secret. However, depending on which environment you are addressing the admin key and secret may differ. Can you provide some additional details on the endpoint you are calling when you encounter this error including the header/body parameters in the request and if you are using the admin key and secret from the original or target environment? If you are concerned with providing these details in a public forum you are welcome to reach out to support instead with these details.

Nuruddin
7 - Meteor

Hi Jeff,

What needs to be done in order to publish workflows to the gallery from the local machine?

 

Thanks,

Nuruddin

Nuruddin
7 - Meteor

Hi All,

 

The POST method to publish workflows in prod environment is not working, giving me 401 unauthorized error. But all the get requests are working fine. Please help me solve the issue. Thank you.

 

Regards,

Nuruddin

matthewbechard
7 - Meteor

Nurrudin-

 

I have the same issue.  I can run the entire workflow with my own customization's but I cannot get the post to work.On my end I also have Symantec firing with a suspicious activity notification when I run it, as its not a recognized program.

 

My question is, how do we test the post portion outside of Alteryx just to test it?

 

Matt

Nuruddin
7 - Meteor

Hi Matt,

 

You test the post portion using postman. 

 

Regards,

Nuruddin

Yongcan
7 - Meteor

@Jefypop 

Really informative, thanks.
Based on your approach and code, we built code promotion/roll back solution via admin api. it's not perfect yet due to current API implementation but it works.

 

Also I want to share some to save people's time as I know it's suffering to make API work and build a working end to end solution from prototype.

 

about sourceId, lots people [ @jason-sa ]  get confused including me at first. it's the workflow id at SOURCE gallery, so with a unique ID will also do so target enviormnt only have one copy of workflow.

Below doc explained the details:
https://help.alteryx.com/current/server/workflows-admin-interface
SourceId
This is the Source environment appId of the workflow to be migrated. If a workflow with the same sourceId already exists, this will replace that workflow in the target environment. Otherwise, a new workflow will be generated.
(Send empty string "" if you do not wish to specify an appID)

 

Expecting Admin API get enhanced in year further so we can make our current solution better.

praveenkchukka
5 - Atom

@JeffreyP

I was able to migrate the workflows from our Test to Production system using the POC workflow, however, they are migrated to my Private Studio but not added to Collections they were in Test system. Is that how it works or am i missing something? We want workflows to be added to the Collections that they are in Test system. Thanks!

Praveen

Rahul_Thakur
5 - Atom

Couple of questions while using POST API feature to publish canvas to gallery

 

Question 1:- Does POST API only support .YXZP files? I am unable to publish .YXMC or .YXMD file 

 

Question 2:-  I am unable to publish  workflow to gallery using POST API when argument for 'owner' is a curator. 

Error message 'The user account you are attempting to publish as is not an artisian' 

 

Question 3:- POST API is not able to replace existing workflow on gallery. It creates a  new version when trying to overwrite - resolved 

 

Question 4:- Do you have an API to delete workflow from gallery

 

@ SeanAdams

JamesHa
Alteryx
Alteryx

I had run into some issues in the program.cs file with the serialization not working.  My colleague and I have done some updates to use System.Text.Json.Serialization instead.  Also, we added an if/else statement for the oauthSignatureString to support the different methods used between the calls (GET/POST/PUT)

 

Disclaimer: THIS CODE IS FOR EXAMPLE USE ONLY AND SHOULD NOT BE USED FOR ANY INTERNAL MIGRATION IMPLEMENTATIONS

 

 

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
//using System.Web.Script.Serialization;
using System.Text.Json.Serialization;

/* THIS CODE IS FOR EXAMPLE USE ONLY AND SHOULD NOT BE USED FOR ANY INTERNAL MIGRATION IMPLEMENTATIONS */

namespace WorkflowMigrations
{
    class Program
    {
        static string sourceEnvironmentBaseUrl = "Your source gallery url here";
        static string targetEnvironmentBaseUrl = "Your target gallery url here";

        //SOURCE ENVIRONMENT API KEY/SECRET
        static string oauthConsumerKeySourceEnvironment = "SOURCE ADMIN KEY HERE";
        static string oauthConsumerSecretSourceEnvironment = "SOURCE ADMIN SECRET HERE";

        //TARGET ENVIRONMENT API KEY/SECRET
        static string oauthConsumerKeyTargetEnvironment = "TARGET ADMIN KEY HERE";
        static string oauthConsumerSecretTargetEnvironment = "TARGET ADMIN SECRET HERE";

        static void Main(string[] args)
        {
            var appsReadyToMigrate = GetAppsReadyForMigration(sourceEnvironmentBaseUrl, "SOURCE STUDIO ID HERE, CAN BE MULTIPLE IDS SEPARATED BY A ,");


            foreach (var appReadyToMigrate in appsReadyToMigrate)
            {
                var app = GetApp(appReadyToMigrate.id);
                PublishAppInTargetEnvironment(app, appReadyToMigrate.id);
            }
        }

        static List<AppsReadyToMigrate> GetAppsReadyForMigration(string sourceBaseEnvUrl, string subscriptionIDs)
        {
            WebClient client = new WebClient();
            var oauthRequest = GenerateOauth1Request("GET", "/api/admin/v1/workflows/migratable/", "&subscriptionIds=" + subscriptionIDs, oauthConsumerKeySourceEnvironment, oauthConsumerSecretSourceEnvironment);
            var url = string.Format("{0}/api/admin/v1/workflows/migratable/?subscriptionIds={1}&{2}",
                sourceBaseEnvUrl,
                subscriptionIDs,
                oauthRequest);
            var options = new JsonSerializerOptions
            {
                PropertyNameCaseInsensitive = true,
            };


            var appsReadyForMigration = client.DownloadString(url);
            var austintest = JsonSerializer.Deserialize<List<AppsReadyToMigrate>>(appsReadyForMigration, options);
            return JsonSerializer.Deserialize<List<AppsReadyToMigrate>>(appsReadyForMigration);

        }

        static byte[] GetApp(string appID)
        {
            WebClient client = new WebClient();
            var oauthRequest = GenerateOauth1Request("GET", $"/api/admin/v1/{appID}/package/", string.Empty, oauthConsumerKeySourceEnvironment, oauthConsumerSecretSourceEnvironment);
            var url = string.Format("{0}/api/admin/v1/{1}/package/?{2}",
                sourceEnvironmentBaseUrl,
                appID,
                oauthRequest);
            

            var app = client.DownloadData(url);
            return app;
        }

        static void PublishAppInTargetEnvironment(byte[] app, string appId)
        {
            try
            {
                var oauthRequest = GenerateOauth1Request("POST", "/api/admin/v1/workflows/", string.Empty, oauthConsumerKeyTargetEnvironment, oauthConsumerSecretTargetEnvironment);
                var url = string.Format("{0}/api/admin/v1/workflows/?{1}",
                    targetEnvironmentBaseUrl,
                    oauthRequest);
                

                var formBoundary = Guid.NewGuid().ToString();
                var multipartFormData = GenerateMultipartFormData(formBoundary, app);
                var httpBodyTask = multipartFormData.ReadAsByteArrayAsync();
                httpBodyTask.Wait();
                var httpBody = httpBodyTask.Result;

                WebClient client = new WebClient();
                client.Headers.Add(HttpRequestHeader.ContentType, string.Format("multipart/form-data; boundary=\"{0}\"", formBoundary));
                client.UploadData(url, httpBody);
                ResetAppInSourceEnvironment(appId);
            }
            catch (Exception ex)
            {
                Console.WriteLine("An error occurred while publishing the workflow to the target environment: {0}", ex);
            }
        }

        static HttpResponseMessage ResetAppInSourceEnvironment(string appId)
        {
            WebClient client = new WebClient();
            var oauthRequest = GenerateOauth1Request("PUT", $"/api/admin/v1/workflows/migratable/{appId}/", string.Empty, oauthConsumerKeySourceEnvironment, oauthConsumerSecretSourceEnvironment);
            var url = string.Format("{0}/api/admin/v1/workflows/migratable/{1}/?{2}",
                sourceEnvironmentBaseUrl,
                appId,
                oauthRequest);

            client.UploadString(url, "PUT", string.Empty);
            return null;
        }

        static string GenerateOauth1Request(string httpMethod, string requestURL, string queryParams, string oauthConsumerKey, string oauthConsumerSecret)
        {
            var oauthTimestamp = (Int32)(DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1))).TotalSeconds;
            var oauthSignatureMethod = "HMAC-SHA1";
            var oauthVersion = "1.0";
            var oauthNonce = Guid.NewGuid().ToString();

            var oauthParamsBuilder = new StringBuilder();
            oauthParamsBuilder.Append(string.Format("oauth_consumer_key={0}", oauthConsumerKey));
            oauthParamsBuilder.Append(string.Format("&oauth_nonce={0}", oauthNonce));
            oauthParamsBuilder.Append(string.Format("&oauth_signature_method={0}", oauthSignatureMethod));
            oauthParamsBuilder.Append(string.Format("&oauth_timestamp={0}", oauthTimestamp));
            oauthParamsBuilder.Append(string.Format("&oauth_version={0}", oauthVersion));
            oauthParamsBuilder.Append(queryParams);

            var oauthSigningKey = $"{oauthConsumerSecret}&";
            var oauthSignatureString = string.Empty;
            if (httpMethod != "POST")
            {
                oauthSignatureString = String.Format("{0}&{1}&{2}",
                    httpMethod,
                    Uri.EscapeDataString(sourceEnvironmentBaseUrl + requestURL),
                    Uri.EscapeDataString(oauthParamsBuilder.ToString())
                );
            }
            else
            {
                oauthSignatureString = String.Format("{0}&{1}&{2}",
                    httpMethod,
                    Uri.EscapeDataString(targetEnvironmentBaseUrl + requestURL),
                    Uri.EscapeDataString(oauthParamsBuilder.ToString())
                );
            }
            HMACSHA1 oauthSignatureHasher = new HMACSHA1(Encoding.UTF8.GetBytes(oauthSigningKey));
            var oauthSignature = oauthSignatureHasher.ComputeHash(Encoding.UTF8.GetBytes(oauthSignatureString));

            var oauthRequestBuilder = new StringBuilder();
            oauthRequestBuilder.Append(string.Format("oauth_consumer_key={0}", oauthConsumerKey));
            oauthRequestBuilder.Append(string.Format("&oauth_nonce={0}", oauthNonce));
            oauthRequestBuilder.Append(string.Format("&oauth_signature_method={0}", oauthSignatureMethod));
            oauthRequestBuilder.Append(string.Format("&oauth_timestamp={0}", oauthTimestamp));
            oauthRequestBuilder.Append(string.Format("&oauth_version={0}", oauthVersion));
            oauthRequestBuilder.Append(string.Format("&oauth_signature={0}", Uri.EscapeDataString(Convert.ToBase64String(oauthSignature))));

            return oauthRequestBuilder.ToString();
        }

        static MultipartFormDataContent GenerateMultipartFormData(string boundary, byte[] app)
        {
            MultipartFormDataContent formData = new MultipartFormDataContent(boundary);
            formData.Add(new StreamContent(new MemoryStream(app)), "file", "Foobar.yxzp"); //GIVE THE NEW WORKFLOW A FILENAME
            formData.Add(new StringContent("Foobar"), "name"); //NAME OF THE WORKFLOW
            formData.Add(new StringContent("email@email.com"), "owner"); //OWNER OF THE WORKFLOW. THIS USER/EMAIL MUST EXIST IN THE TARGET SYSTEM
            formData.Add(new StringContent("false"), "validate"); //DO NOT EXECUTE A VALIDATION RUN
            formData.Add(new StringContent("false"), "isPublic"); //DO NOT MAKE THIS A PUBLIC WORKFLOW
            formData.Add(new StringContent(""), "sourceId"); //INSERT THE SOURCE ENVIRONMENT AppID OF THE WORKFLOW IN THE TARGET ENVIRONMENT. 
                                                             //IF A WORKFLOW WITH THE SAME SOURCE ID ALREADY EXISTS THEN THIS WILL REPLACE THAT WORKFLOW IN THE TARGET ENVIRONMENT, OTHERWISE A NEW WORKFLOW WILL BE GENERATED
            formData.Add(new StringContent("foo"), "workerTag"); //ADD A WORKER TAG TO THE WORKFLOW
            formData.Add(new StringContent("true"), "canDownload"); //USERS CAN DOWNLOAD THE WORKFLOW

            return formData;
        }
    }

    public class AppsReadyToMigrate
    {
        public string id { get; set; }
        public string publishedVersionId { get; set; }
        public string subscriptionId { get; set; }
    }
}

 

 

nehachopade
5 - Atom

Hi Jeffrey,   We have couple of thousands Alteryx workflows scheduled in on-premise solution. We need to put all these workflows to Alteryx which is installed on AWS. On-premise and AWS has Alteryx version 2020.2 Could you please guide me on to the best possible way though which this can be done? and what all factors needs to be considered during this activity?

KevinP
Alteryx
Alteryx

@nehachopade The API's referenced here can by used to move selected workflows from one environment to another. There is also some information related to this in the online help at: https://help.alteryx.com/current/server/workflows-admin-interface (reference the Workflow Migration section)


If you need additional assistance implementing or utilizing these API's and the workflow migration process I would recommend reaching out to our Support team by opening a support ticket, or engaging a Solutions Architect by talking with your account representative.

PiotrGa
6 - Meteoroid

Hi Jeffrey,

very helpful article and examples, thank you.

 

I am trying to adopt the example you attached however once it comes to publishing the EXE supporting app returns me an error 403. It means that the access is forbidden. I have admin rights on both source and target galleries and  admin API is enabled (I see key & secret). Also I input my email and it looks I authenticate so server recognize my credentials.

 

Is there any thing I am missing or any settings that must be changed to allow me to publish workflows through API.

 

Thanks for any clue

Piotr

David-Carnes
11 - Bolide

The migratable endpoint appears to disregard the Gallery Data Connections on the destination server.  As a test I used the workflow provided here and pointed at the same server for both getting the workflow and migrating the workflow.  I even tried setting the app ID to an empty string, to create a new workflow, in addition to using the existing app ID to overwrite the existing workflow.

 

If no Gallery Data Connection is present, the workflow can be migrate.  But we have moved over to 100% Gallery Data Connections for all our database connections but now the POST to api/admin/v1/workflows/ is a useless endpoint.

 

I have started a ticket with Support and also added a new Idea: Enhance the migratable endpoint to validate Gallery Data Connections in YXZPs 

 

A similar Idea is @SeanAdams's Server API to extract / submit workflows 

 

Please go to these Ideas and upvote them.

 

Thank you for your time.

Sincerely,

David