(Guide) Developing Custom Tools for Designer

mkhtran

This is (yet another) guide for developing custom tools in Alteryx Designer. My hope is to bring more attention to the development process which I have found to be extremely frustrating, by sharing what I have found to be extremely useful during my experience. The focus is on helpful things not mentioned in most other Alteryx guides I've found, while focusing on Alteryx-specific oddities.

I am not affiliated with Alteryx. My advice here is only given as a 3rd-party developer working with the SDK for a while. Any suggestions or corrections are highly appreciated!

Prerequisites

• Python v3.8 - https://www.python.org/downloads/release/python-3810/
• Node.js v16.20 - https://nodejs.org/download/release/v16.20.1/
• pip and npm (the above should install these)

These are the highest versions I have found to work with the SDK. Alteryx recommends Node v14.17 and npm v6.14, but I ran into more TypeScript issues with those versions. Node v16.20 will come with npm v9+ which will require the `--legacy-peer-deps` flag for most commands to work with the SDK. (I do not use conda. Sorry!)

Then install the Alteryx tools.

> pip install ayx-plugin-cli
> pip install ayx-python-sdk

> ayx_plugin_cli version
Alteryx CLI Version: 1.1.0

Initializing the Workspace

Create a new folder for the workspace and initialize it.

> mkdir MyPlugins
> cd MyPlugins
> ayx_plugin_cli sdk-workspace-init

The `sdk-workspace-init` command should prompt you for settings. This can also later be edited in `ayx_workspace.json`.

Initializing the Plugin

Create a new plugin.

> ayx_plugin_cli create-ayx-plugin --use-ui

The `create-ayx-plugin` command should prompt you for settings. Make sure that `--use-ui` is enabled. Newer versions of `ayx_plugin_cli` do not enable this by default.

Update UI dependencies in `\ui\ToolName\package.json` to the following (or most recent) versions:

"dependencies": {
  "@alteryx/icons": "^1.2.0",
  "@alteryx/react-comms": "^1.0.1",
  "@alteryx/ui": "^1.4.5",
}

This is a very important step. The Alteryx SDK has not been updated to use the most recent versions yet which most documentation will refer to. The older versions are also very buggy. Remember that the `--legacy-peer-deps` flag is required in `npm install --legacy-peer-deps`.

I recommend the following changes as well:

• Change the `build` script for the UI component to be "npm install --legacy-peer-deps && webpack --config webpack.prod.js". Adding `npm install` here may be necessary depending on your build pipeline.
• Add the following patterns to your version control system's ignore:
  • node_modules - Created when developing the UI
  • .ayx_cli.cache, build, dist, PlaceholderToolName - Created by `ayx_plugin_cli` during certain actions (I'm not sure why PlaceholderToolName keeps being created)
  • __pycache__, .pytest_cache - Created when developing and testing the backend
• Configure TypeScript as desired.

Building the Plugin

While the `create-yxi` and `designer-install` scripts work well, I've added the following scripts to fix some issues.

"scripts": {
  "build": "npm install && npm run ayx-build",
  "test": "python -m pytest backend",
  "ayx-build": "rd /s /q \"ui\\PlaceholderToolName\" & ayx_plugin_cli create-yxi --use-ui",
  "ayx-build-omit-ui": "rd /s /q \"ui\\PlaceholderToolName\" & ayx_plugin_cli create-yxi",
  "ayx-install": "rd /s /q \"ui\\PlaceholderToolName\" & echo y | ayx_plugin_cli designer-install --use-ui --install-type user",
  "ayx-install-omit-ui": "rd /s /q \"ui\\PlaceholderToolName\" & echo y | ayx_plugin_cli designer-install --install-type user"
}

The `ayx_plugin_cli` seems to create `ui\PlaceholderToolName` when building, and if this folder exists already, it can actually cause future builds to fail. The `ayx-build-omit-ui` and `ayx-install-omit-ui` variants simply skip rebuilding the UI when building which can often speed up the process immensely if you are not modifying the UI. The `build` script for each tool should also be modified to include `npm install` as described in the previous section, otherwise the build can fail.

This creates the `build\yxi\PluginName.yxi` file that you can distribute.

Note that `create-yxi` will call `npm run build` on each of your tools to create the UI. If this fails (mismatched Node or TypeScript versions can cause building here to fail where it didn't fail during development), then a horribly formatted flurry of errors and stack traces will be printed to the screen. You should run `npm run build` yourself to see the exact error formatted correctly.

Developing the UI

All the UI development should be done utilizing Alteryx's `dev-harness` to enable previewing and Hot Reloading without having to rebuild and reinstall the entire plugin.

Start the Webpack server.

> cd ui\ToolName
> npm start

The `start` script starts a Webpack DevServer on port 8080 and automatically opens `localhost:8080`. If you want to use a different port (e.g. if you want to develop multiple tools at once), you will need up make changes to the `dev-harness` as described here: https://github.com/alteryx/dev-harness/issues/11

The entry point to the UI is `ui\ToolName\src\index.tsx`. The key structure of the UI piece is

<DesignerApi
  messages={...}
  defaultConfig={...}
>
  <AyxAppWrapper>
    <App />
  </AyxAppWrapper>
</DesignerApi>

The `DesignerApi` and `AyxAppWrapper` components manage communication between your UI and the Alteryx layer. For example, it provides the model context. (See https://alteryx.github.io/react-comms)

import { Context as UiSdkContext } from '@alteryx/react-comms'

const App = (): JSX.Element => {
  const [model, handleUpdateModel] = useContext(UiSdkContext)
  /* ... */
}

Developing the Backend

The backend is mostly straightforward, aside from the lack of documentation. There are plenty of resources and examples regarding a tool's lifecycle, managing metadata, and understanding anchors/connections/batches.

Configuration

The main purpose of the UI is to manage configuring your tool. The configuration begins in the `defaultConfig` prop of the `DesignerApi` component.

<DesignerApi
  messages={{}}
  defaultConfig={{
    Configuration: {
      foo: 'Hello, world!',
      bar: 62
    }
  }}
>
  {/* ... */}
</DesignerApi>

These values can then be accessed from the model.

const [model, handleUpdateModel] = useContext(UiSdkContext)

const foo = model.Configuration.foo
const bar = model.Configuration.bar

Updating these values are done as such:

const App = (): JSX.Element => {
  const [model, handleUpdateModel] = useContext(UiSdkContext)

  const handleChangeFoo = (event: ChangeEvent<HTMLInputElement>): void => {
    handleUpdateModel({
      ...model,
      Configuration: {
        ...model.Configuration,
        foo: event.target.value
      }
    })
  }

  return (
    <TextField value={model.Configuration.foo} onChange={handleChangeFoo} />
  )
}

Note that the configuration is only saved when `handleUpdateModel()` is called. If it is never called, then not even the default configuration will be saved for the backend to later access. I recommended adding this hook to handle that case:

const App = (): JSX.Element => {
  const [model, handleUpdateModel] = useContext(UiSdkContext)

  useEffect(() => {
    handleUpdateModel({ ...model })
  }, [])

  /* ... */
}

Since the configuration has to be serialized and deserialized in the JavaScript UI layer, to the Python backend, and through XML, etc., I've found it most reliable to always have your configuration values be strings. I set everything as JSON-strings.

First update `defaultConfiguration` to make sure every value is a JSON string.

function createConfiguration<C extends Record<string, any>> (config: C): { [K in keyof C]: string } {
  const newConfig: { [K in keyof C]: string } = {} as any
  for (const key in config) {
    newConfig[key] = JSON.stringify(config[key])
  }
  return newConfig
}

const defaultConfiguration = createConfiguration({
  foo: 'Hello, world!',
  bar: 62
})

<DesignerApi
  messages={{}}
  defaultConfig={{
    Configuration: defaultConfiguration
  }}
>
  {/* ... */}
</DesignerApi>

Then here's a dandy `useConfiguration()` hook that will allow you to manage configuration values just like `useState()`, and have it be type-aware. (Note that you cannot use more than one `setValue()` call in a single tick. This is discussed in the Annotation section.)

function useConfiguration<T> (key: string): [T, (value: T) => void] {
  const [model, handleUpdateModel] = useContext(UiSdkContext)
  const value = JSON.parse(model.Configuration[key]) as T
  const setValue = (value: T): void => {
    handleUpdateModel({
      ...model,
      Configuration: {
        ...model.Configuration,
        [key]: JSON.stringify(value)
      }
    })
  }
  return [value, setValue]
}

const [foo, setFoo] = useConfiguration<string>('foo')
const [bar, setBar] = useConfiguration<number>('bar')

setBar(bar + 1)

You have to deserialize them in the backend too

self.foo: str = json.loads(self.provider.tool_config.get("foo"))
self.bar: int = json.loads(self.provider.tool_config.get("bar"))

Secrets

The Secrets section of the model works similarly to the Configuration, but values stored here are encrypted/obfuscated when saved. This is useful for sensitive configuration values like passwords, but I have not yet figured out how access Secrets in the backend, making its usefulness limited. Also note that Secrets will not behave properly in the `dev-harness`, so you may have to test it inside of Designer.

I use Secrets just like how I use the Configuration section.

type AyxEncryptionMode = 'obfuscation' | 'machine' | 'user'
function createSecrets<S extends Record<string, { value: any, encryptionMode?: AyxEncryptionMode }>> (secrets: S): { [K in keyof S]: { text: string, encryptionMode: string } } {
  const newSecrets: { [K in keyof S]: { text: string, encryptionMode: string } } = {} as any
  for (const key in secrets) {
    newSecrets[key] = {
      text: JSON.stringify(secrets[key].value),
      encryptionMode: secrets[key].encryptionMode ?? 'obfuscation'
    }
  }
  return newSecrets
}

const defaultSecrets = createSecrets({
  password: { value: 'password123' }
})

<DesignerApi
  messages={{}}
  defaultConfig={{
    Configuration: defaultConfiguration,
    Secrets: defaultSecrets
  }}
>
  {/* ... */}
</DesignerApi>

And its own `useSecrets()` hook:

function useSecrets<T> (key: string): [T, (value: T) => void] {
  const [model, handleUpdateModel] = useContext(UiSdkContext)
  const value = JSON.parse(model.Secrets[key].text) as T
  const setValue = (value: T): void => {
    handleUpdateModel({
      ...model,
      Secrets: {
        ...model.Secrets,
        [key]: {
          ...model.Secrets[key],
          text: JSON.stringify(value)
        }
      }
    }
  }
  return [value, setValue]
}

const [password, setPassword] = useSecrets<string>('password')

Annotation

Handling the tool's annotation is a bit convoluted. The main annoyance is that Alteryx doesn't actually populate `model.Annotation` like it does with the Configuration. This means that you always have to set the Annotation correctly even if nothing changes in the configuration. This ends up being extremely problematic if your configuration logic is split across multiple files.

The best way I found to reliably do this is support batching updates to the model. Here's a basic example where the `useContext(UiSdkContext)` hook is replaced with a `useModel()` hook that supports batched updates with the Spec syntax:

import update from 'immutability-helper'
import type { Spec } from 'immutability-helper'

function useModel (): [any, ($spec: Spec<any>, flushImmediately?: boolean) => void] {
  const [model, handleUpdateModel] = useContext(UiSdkContext)

  const flushBatch = (): void => {
    const newModel = updateBatch.reduce((newModel, $spec) => update(newModel, $spec), model)
    updateBatch = []
    updateId = null
    handleUpdateModel(newModel)
  }

  const updateModel = ($spec: Spec<any>, flushImmediately: boolean = false): void => {
    updateBatch.push($spec)
    if (updateId === null) {
      updateId = setTimeout(flushBatch)
    }

    if (flushImmediately) {
      clearTimeout(updateId)
      flushBatch()
    }
  }

  return [model, updateModel]
}

let updateBatch: any[] = []
let updateId: Timeout | null = null

Once the `useConfiguration()` and `useSecrets()` hooks are updated to use the new `useModel()` hook, then you no longer have to worry about subsequent changes to the model overwriting previous calls. The Annotation can now be handled in a single `useEffect()` hook like so:

const DEVELOPMENT = process.env.NODE_ENV == null || process.env.NODE_ENV === 'development'

const App = (): JSX.Element => {
  const [model, updateModel] = useModel()
  const [foo, setFoo] = useConfiguration<string>('foo')

  useEffect(() => {
    if (DEVELOPMENT) {
      return
    }

    updateModel({
      Annotation: { $set: `Foo is "${foo}"` }
    })
  }, [model.Configuration])

  /* ... */
}

Note that the annotation is not updated in development. The `dev-harness` is currently a little buggy when it comes to how it updates the model, and I highly recommend only trying to update `model.Configuration` in development. Updating other parts of the model can cause strange behavior (including infinite render loops).

JsEvent

The `async JsEvent()` function is Alteryx's best kept secret. This function allows you to access Designer's C# layer to do things the UI alone can't accomplish, or to get information only Alteryx knows. Unfortunately this method has essentially no official documentation, and everything has to be figured out through trial-and-error. Here's the basic syntax:

const result = await JsEvent(
  'FileBrowse',                                            // Event name
  { browseType: 'File', fileTypeFilters: 'All files|*.*' } // Event-specific parameters
)

Note that this method seems to fail silently if the wrong parameters are provided, and a number of events will never even resolve when used correctly. Here are some useful ones I've discovered (all my findings can be seen at https://github.com/alteryx/alteryx-ui/issues/25#issuecomment-1721851391):

JsEvent	Arguments	Returns	Description
GetInputData	{ anchorIndex: number, connectionName: string, numRecs?: number, offset?: number }	{ fields: Array<{ strName: string, strType: AyxDataType, nSize: string, strSource: string, strDescription: string }>, numRecs: number, data: string[][] }	Get data from any input connections.
FileBrowse	{ browseType?: 'File' | 'Folder', startingDirectory?: string, fileTypeFilters?: string, defaultExtensionChoice?: unknown }	string	Open a file explorer window and get the path of the selected file or directory.
GetFileContents	{ Path: string }	string	Get the contents of a file.

Localization

Localization is pretty straightforward, and is done through the `react-intl` package. Create a `ui\ToolName\lang` folder that will hold all the translation files. The supported locales are en, zh, fr, de, it, ja, pt, es.

// lang\en.json
{
  "greeting": "Hello, world!"
}

// lang\fr.json
{
  "greeting": "« Bonjour le monde »"
}

These need to be passed to `DesignerApi`'s `messages` prop.

import en from '../lang/en.json'
import fr from '../lang/fr.json'

const messages = { en, fr }

<DesignerApi
  messages={messages}
  defaultConfig={...}
>
  {/* ... */}
</DesignerApi>

Within the components, you can use `react-intl` to translate messages.

import { FormattedMessage, useIntl } from 'react-intl'

const intl = useIntl()
console.log(intl.formatMessage({ id: 'greeting' }))

return <FormattedMessage id='greeting' />

Note that `dev-harness` does not yet support testing locales. If you want to enable it, check out this issue here: https://github.com/alteryx/dev-harness/issues/15

Model

While Alteryx does define an `IModel` interface that the model should adhere to, I've found it to be insufficient. I use the following typings:

type AyxEncryptionMode = 'obfuscation' | 'machine' | 'user'

type AyxModel<C extends Record<string, any>, S extends Record<string, { value: any, encryptionMode?: AyxEncryptionMode }>> = {
  Annotation?: string
  Configuration: AyxConfiguration<C>
  Meta?: unknown[] | FieldListArray
  Secrets: AyxSecrets<S>
  ToolId: number
  ToolName: string
  srcData: unknown
}

type AyxConfiguration<T extends Record<string, any>> = {
  [K in keyof T]: string
}

type AyxSecrets<T extends Record<string, { value: any, encryptionMode?: AyxEncryptionMode }>> = {
  [K in keyof T]: { text: string, encryptionMode: AyxEncryptionMode }
}

Notably, `model.Annotation` is always `undefined` until you set it yourself. `model.Meta` can also take on different shapes at certain stages of the tool's lifecycle. It can be `undefined`, an array of fields, or an expanded version of an array of fields (FieldListArray).

For rock-solid typings in your project, you can then follow this paradigm:

import type { AyxModel } from '...'

export function useModel<C extends Record<string, any>, S extends Record<string, any>> (): [AyxModel<C, S>, ($spec: Spec<AyxModel<C, S>>, flushImmediately?: boolean) => void] {
  /* ... */
}

export function useConfiguration<T> (key: string): [T, (value: T, flushImmediately?: boolean) => void] {
  const [model, updateModel] = useModel<Record<string, T>, Record<string, unknown>>()
  /* ... */
}

export function useSecrets<T> (key: string): [T, (value: T, flushImmediately?: boolean) => void] {
  const [model, updateModel] = useModel<Record<string, unknown>, Record<string, { value: T }>>()
  /* ... */
}

import type { AyxModel } from '...'
import {
  useConfiguration as baseUseConfiguration,
  useModel as baseUseModel,
  useSecrets as baseUseSecrets
} from '...'
import type { Spec } from 'immutability-helper'

export const defaultConfiguration = {
  foo: 'Hello, world!',
  bar: 62
}

export const defaultSecrets = {
  password: { value: 'password123' }
}

export type MyConfiguration = typeof defaultConfiguration
export type MySecrets = typeof defaultSecrets
export type MyModel = AyxModel<MyConfiguration, MySecrets>

export function useModel (): [MyModel, ($spec: Spec<MyModel>, flushImmediately?: boolean) => void] {
  return baseUseModel<MyConfiguration, MySecrets>()
}

export function useConfiguration<K extends keyof MyConfiguration> (key: K): [MyConfiguration[K], (value: MyConfiguration[K], flushImmediately?: boolean) => void] {
  return baseUseConfiguration<MyConfiguration[K]>(key)
}

export function useSecrets<K extends keyof MySecrets> (key: K): [MySecrets[K]['value'], (value: MySecrets[K]['value'], flushImmediately?: boolean) => void] {
  return baseUseSecrets<MySecrets[K]['value']>(key)
}

This fully supports type-inference when accessing the Configuration.

// The full shape of `model` is already determined
const [model, updateModel] = useModel()

// `foo` is known to be a string
const [foo, setFoo] = useConfiguration('foo')

// Errors on nonexistent configuration keys
const [bar, setBar] = useConfiguration('not-a-valid-key')

Missing Features

The following aspects of custom plugins do not yet seem to be supported by the SDK.

• Folder Icon - There is no way to provide an icon for the Tool Category for your plugin. You can add an image file in `C:\Program Files\Alteryx\bin\RuntimeData\icons\categories` with the exact name of the Tool Category.
• Workflow Examples - There is no way to package example workflows with your tool. It seems you must add an example workflow in `Program Files\Alteryx\Samples\[locale]\` and set the `Metadata > Example` config of your tool to be
  

<Example>
	<Description>Open Example</Description>
	<File>MyPlugin\ExampleWorkflow.yxmd</File>
</Example>

• Help URL - There is the undocumented `tools.[plugin].configuration.help_url` option in `ayx_workspace.json`. This seems to work best for URLs, and only when Help > Source setting is set to "Online". It does not work otherwise, and there is no way to package local help files with your tool.

mkhtran

Help URL

I've found some additional information regarding the Help URL configuration.

• Except for help, all paths are relative to this config file. The variables [Engine.EngineDirectory], [Engine.PluginDirectory] and [Engine.JavaScriptPluginDirectory] can also be used.
• There are five valid settings for the Help attribute of the GuiSettings tag.
• No Help attribute, or empty quotes, will take the user to the Alteryx web help index. This is the default behavior.
• A valid HTTP URL including http:// or https:// protocol will open the default browser to that URL.
• A filename of the form file://path/FileName.html will load the specified file from this plugin's directory.
• A filename of the form file://C:/path/FileName.html will load the specified file from an absolute path.
• An HTML filename without a preceding path or protocol indicator will load that HTML file in the Alteryx web help.

If your help URL should link to a webpage, then configuration is straightforward. If you decide to package your help files as local files (e.g. a local .pdf or local .html file), then you can use the fact that a filename of the form file://path/FileName.html is relative to the plugin's directory.

This means that you can package a local help file in the following way:

1. Create the help file that will be packaged. For example `help.pdf`.
2. Set the Help URL configuration of your tool to `file://help.pdf`.
3. Build the plugin's .yxi installer as usual.
4. Append the `help.pdf` file to the .yxi's archive at `[ToolName]\help.pdf`.

The installer will now install the `help.pdf` file to the tool's installation folder and the help URL will point to it.

Here's an example Node.js script that injects the help file as above.

const AdmZip = require('adm-zip')
const fs = require('fs')
const path = require('path')

const yxi = path.resolve(__dirname, './build/yxi/MyPlugins.yxi')
const helpFile = path.resolve(__dirname, './help.pdf')

const zip = new AdmZip(yxi)
zip.addLocalFile(helpFile, `${TOOL_NAME}/help.pdf`)
zip.writeZip(yxi)

This still does not support locales or offline mode. It is assumed that Alteryx has no support for this.