Server API: Download packages for surveilance

Question

Hi all,

We are building out a process to download all canvasses from our server to be able to inspect and auto-check for issues (like embedded passwords, tools that may be misconfigured or potentially harmful and needing further review etc).

4 questions (first 2 are most pressing)

- The current API for downloading a package (/gallery/api/admin/v1/{appid}/package downloads these packages in a zipped format.     Can you point me to an API where we can get these in plain XML format so that we don't need to first dump to a file; unzip; and then extract?.

- If not - can you direct me to a method to unpack these within the data stream rather than saving to file (e.g. possibly using something like python)?

- Is there some way to read this directly from the underlying DB instead (again, in a plain XML readable format)?

- Where can I find a document that contains the full server API and data dictionary, similar to this:

* https://onlinehelp.tableau.com/current/server/en-us/data_dictionary.html
* https://onlinehelp.tableau.com/current/api/rest_api/en-us/help.htm#REST/rest_api.htm%3FTocPath%3D_____1
* https://community.lithium.com/t5/Developer-Documentation/bd-p/dev-doc-portal?section=commv2

Thank you

Sean

cc: @HeatherMHarris @revathi @LizaNemchynova @AshwiniChezhiyan @JulieM

KevinP · Accepted Answer

@SeanAdams Thank you for your post and I will do my best to answer each of your questions. Please see my feedback on your questions below in red.

* The current API for downloading a package (/gallery/api/admin/v1/{appid}/package downloads these packages in a zipped format. Can you point me to an API where we can get these in plain XML format so that we don't need to first dump to a file; unzip; and then extract? The package endpoint for the Gallery Admin API provides the workflow as a packaged yxzp file. It isn't possible to extract the raw XML via the API. This is because there isn't an endpoint function available to do so, and due to the way these workflows are stored in the database. In order to provide the raw xml the API would have to retrieve and reassemble the binary chunks for the package, store that package in a temp location, extract the package, identify the correct file from the package (packages can contain multiple workflows/apps/macros), read that file as raw text, and provide that xml text stream as output.
* If not - can you direct me to a method to unpack these within the data stream rather than saving to file (e.g. possibly using something like python)? The method to unpack the package would really depend on what tools you are using to develop this process. There are posts on StackOverflow that cover how to extract zip files using python (https://stackoverflow.com/questions/3451111/unzipping-files-in-python) and other languages. If you are building the process in Alteryx Designer you can retrieve the yxzp from the api as a blob output. That blob can then be saved to a file, and you could then use a Run Command tool to execute a script to extract the zip file.
* Is there some way to read this directly from the underlying DB instead (again, in a plain XML readable format)? No, due to the way the workflows are stored in the database as binary chunks it isn't possible to retrieve them directly from the database, and we don't offer any tools to do so outside of the Gallery admin API. Even if you could retrieve them directly they would be in a packaged binary format and not raw XML.
* Where can I find a document that contains the full server API and data dictionary, similar to this: The full interactive documentation for the Gallery API is available from any Alteryx Server. This documentation can typically be found at http(s)://YourServerName.tld/gallery/api-docs. This documentation is also available from our public gallery at https://gallery.alteryx.com/api-docs but you won't be able to interact with the public documentation unless you have an API key for our public gallery. As for schema information (Data Dictionary) for the underlying MongoDB I am not aware of any publicly available documentation regarding this schema. Support does have some limited documentation on the schema that can be provided on request, but this documentation only covers select collections in the database that maybe useful for reporting purposes. It does not contain any information on the storage of any workflow packages or any potentially sensitive data.

SeanAdams · Accepted Answer

@KevinP - firstly, thank you for the detailed response!

@KevinP wrote:@SeanAdams Thank you for your post and I will do my best to answer each of your questions. Please see my feedback on your questions below in red.

* The current API for downloading a package (/gallery/api/admin/v1/{appid}/package downloads these packages in a zipped format. Can you point me to an API where we can get these in plain XML format so that we don't need to first dump to a file; unzip; and then extract? The package endpoint for the Gallery Admin API provides the workflow as a packaged yxzp file. It isn't possible to extract the raw XML via the API. This is because there isn't an endpoint function available to do so, and due to the way these workflows are stored in the database. In order to provide the raw xml the API would have to retrieve and reassemble the binary chunks for the package, store that package in a temp location, extract the package, identify the correct file from the package (packages can contain multiple workflows/apps/macros), read that file as raw text, and provide that xml text stream as output.
* If not - can you direct me to a method to unpack these within the data stream rather than saving to file (e.g. possibly using something like python)? The method to unpack the package would really depend on what tools you are using to develop this process. There are posts on StackOverflow that cover how to extract zip files using python (https://stackoverflow.com/questions/3451111/unzipping-files-in-python) and other languages. If you are building the process in Alteryx Designer you can retrieve the yxzp from the api as a blob output. That blob can then be saved to a file, and you could then use a Run Command tool to execute a script to extract the zip file.

Given that this would be a common need for many server administrators, I'll submit this as an idea in the server Ideas section.    Rather than asking each server admin to go through the trouble independantly of having to learn out to programatically extract and unzip their canvasses - it makes more sense for this to become a standard server API end-point (since Alteryx Server already knows how to unpack these packages for execution).

* Where can I find a document that contains the full server API and data dictionary, similar to this: The full interactive documentation for the Gallery API is available from any Alteryx Server. This documentation can typically be found at http(s)://YourServerName.tld/gallery/api-docs. This documentation is also available from our public gallery at https://gallery.alteryx.com/api-docs but you won't be able to interact with the public documentation unless you have an API key for our public gallery. As for schema information (Data Dictionary) for the underlying MongoDB I am not aware of any publicly available documentation regarding this schema. Support does have some limited documentation on the schema that can be provided on request, but this documentation only covers select collections in the database that maybe useful for reporting purposes. It does not contain any information on the storage of any workflow packages or any potentially sensitive data.

For the API documentation - the challenge that we're facing is that the documentation is not yet detailed enough to be able to be able to quickly build against.

As an example - the API documentation (https://gallery.alteryx.com/api-docs) does not contain any Schema definition for the responses; detail of the meaning of different fields; examples of different use cases; changing between XML and JSON response types etc.

Compare this to another API example - in this case Lithium:   https://community.lithium.com/t5/Developer-Documentation/bd-p/dev-doc-portal?section=commv2   They have a very well documented API which allows people to very quickly become productive

I'll submit this as an idea to the community too since I think there is merit in having a dedicated set of navigable pages that describe the API in depth; along with the execution path that each API takes (for example the API goes to the gallery; which then goes to the controller; then to the first available worker since an API call to run a canvas gets pushed to the front of the queue; etc)

For the database documentation - this is an area that would be valuable to publish - again in a navigable format, since administrators need to use information about shedule runs; errors; etc - to be able to manage their servers.

Thank you Kevin

Sean