How is the engine chosen for a particular workflow

[Paul_Holden]

Hi,

We are upgrading to 2022.1 and I need to understand more about the changes to the Engine options.

I understand the server documentation (I think) and the options available for each server node. What I am failing to find out is what controls the choice of engine for each workflow and how that may, or may not, change with the upgrade i.e. which engine will be chosen for any particular workflow.

Previously I don't /think/ that we had any workflows that needed the AMP engine, so initially I selected Original only for our 2022.1 UAT system, however I am now being asked to switch AMP on because of a specific requirement for Exchange online tools.

I don't fully understand the implications of this change now that AMP on is a "default" in 2022.1

e.g.

If a workflow ran on the Original engine before the upgrade will it definitely continue to run on the Original engine afterwards i.e. do we need to do anything to ensure that workflows continue to run as they have previously using the Original engine?

If we have a workflow running in Production (on 2020.4) which is running on the Original engine and a developer publishes a copy fresh to our UAT (on 2022.1) do they need to do something to force it to run on the Original engine i.e. in order to make the test run on the same engine as currently in Prod?

Apologies if this is RTFM but since we haven't used the AMP engine before I'm really not clear on the documentation in this area and I can't see it linked from the server docs.

Regards,
Paul

[TonyaS · Accepted solution]

Hello @Paul_Holden , 

Thanks for your interest in AMP. With 2022.1 AMP was made the default for NEW workflows that users are building in 2022.1. 

To answer your 2 specific questions:

If a workflow ran on the Original engine before the upgrade will it definitely continue to run on the Original engine afterwards i.e. do we need to do anything to ensure that workflows continue to run as they have previously using the Original engine? workflows would continue to run with whatever Engine setting the workflow was saved with. 

If we have a workflow running in Production (on 2020.4) which is running on the Original engine and a developer publishes a copy fresh to our UAT (on 2022.1) do they need to do something to force it to run on the Original engine i.e. in order to make the test run on the same engine as currently in Prod? Nothing additional is needed, the workflow will always run respecting the Engine setting specified in the workflow. Before 2022.1 the "Use AMP Engine" checkbox was not checked by default, so if a user went in and checked that and saved the workflow they could have AMP enabled.

In all supported versions prior to 2022.1, users can enable AMP by checking the "Use AMP Engine" checkbox in the Workflow Configuration - Runtime settings tab. 

When you save a workflow with "Use AMP Engine" enabled, it will run with AMP anytime it is run unless that setting is changed. 

For Server it will attempt to run with AMP, but if the Server does not support AMP then users will receive an error and either the workflow needs to be saved without AMP enabled or the Server Admin needs to enable AMP on the Server Instance. 

Here are some docs about Server settings related to AMP: 

https://help.alteryx.com/current/server/controller#configuration

https://help.alteryx.com/current/server/worker

https://help.alteryx.com/current/server/engine

https://help.alteryx.com/20221/server/amp-engine-best-practices

If you are going to migrate workflows that were previously built using the original Engine to AMP Engine, you may find the Engine Compatibility Mode helpful - this maintains the record order of the original Engine by using the same sort method. There is a slight performance cost, but AMP should still perform faster than the original Engine. 

Here are the 2022.1 Release notes: https://help.alteryx.com/release-notes/designer/designer-20221-release-notes
And explanation of the change to have AMP as the default Engine for new workflows details:

AMP as Default Engine

AMP is now the default Engine for any new workflows you create. However, you can still choose the engine for your workflows via the workflow-level setting. Go to the Workflow - Configuration pane in Designer and select the Runtime tab. Next, enable or disable the Use AMP Engine check box to make your engine selection.

Please note that when you launch the 22.1 version of Designer, you’ll be greeted with a pop-up to notify you that AMP is the default for new workflows. You can then select to not show the message again if you choose.

Alteryx Multi-threaded Processing (AMP) is a new architecture of Alteryx Engine, designed to work with larger volumes of data at a higher velocity. To learn more about the AMP Engine, as well as the differences between the original engine and AMP, visit these resources:

• Alteryx AMP Engine
• Alteryx Engine and AMP: Main Differences
• Tool Use with AMP
• AMP Memory Use

AMP Engine supports Standard, Iterative, and Batch Macros. In addition, over 85 tools have been converted to run with AMP. Tools that haven’t been converted will automatically use the original Engine.

Field Limit for Input and Output Data with Original Engine and AMP

For Input and Output data, there is now a 32,000 upper limit of fields per record for both the original Engine and AMP.

• When read/write data is less than or equal to 32,000 fields, a valid workflow runs successfully.
• When read/write data exceeds 32,000 fields, an error is returned on a workflow run to indicate that the max number of fields was exceeded: "Error - Number of fields is larger than the 32,000 limit. Please reduce the number of fields."

AMP Grouping Framework Changes

The grouping framework on AMP Engine has been changed to behave similarly to the original Engine. This affects the Summarize, Join, and Join Multiple tools.

[Paul_Holden]

@TonyaS thank you so much for the most complete answer to a question I think I have ever seen on a community board.

One thing that I missed originally that I'll call out is that as well as setting "Engine:" in Engine > General to "Both Engines" you do need to also check the "Engine: Enable AMP Engine" box in Controller > General 

That's down to me not checking all the documentation correctly but I /think/ it is an easy one to miss initially.

Thanks again,

Paul