Engine Works
Under the hood of Alteryx: tips, tricks and how-tos.
Alteryx made its AMP (Alteryx Multi-threaded Processing) Engine the default engine for new workflows in Designer with the release of version 2022.1 in May 2022. On Server, new Installations or additions of worker nodes will enable both AMP and original Engine execution. AMP is Alteryx’s newer generation engine designed to work with larger volumes of data coming in at higher velocity and typically executes workflows faster, with more efficient use of machine resources. This allows execution times with Alteryx to stay ahead of the changing data landscape, where data generation is getting faster, larger, and growing in sources.
AMP Engine processes data parallelly in data packets to leverage the underlying hardware to the maximum. This parallelization using data packets can provide performance gains, especially in larger data sets or on hardware that is able to leverage its multi-threaded architecture. Gains may vary due to environmental and usage variables. As multi-threading forms the core of AMP, it fundamentally takes a different approach to handle data as compared to the ‘original Engine’ (E1), which was a single-threaded engine. This sometimes can result in a different order of output records as AMP uses hashing and binning methodology for sorting and grouping. To address differences in output order, the 'Engine compatibility' mode was introduced, which solves many of such use cases.
AMP sees strong endorsement from our customers, and to further instill user confidence with the adoption, we have recently released “Engines Parity Utility” in our Community Gallery. This utility is available starting with version 22.3. It allows you to compare workflow results using both the 'original Engine' and 'AMP Engine.' This is phase one of the utility, and in phase 2, we aim to provide the ability to convert workflows to AMP once the user is satisfied with the result.
What is the Engines Parity Utility?
Engines Parity Utility is a framework that is made available in the form of a macro tool and an analytic app to Alteryx users who want to compare their workflow execution with the two Alteryx native engines (original Engine and AMP). The tool allows the comparison of multiple workflows in a single run, so users don't have to run each workflow individually. You can point the tool/app to the directory containing workflow/workflows. The output from the run is a report that provides insight into the execution, such as if the results were a match, which engine was faster, and by how much. The output also provides a list of tool IDs in the workflow that produced different results. You can also look at the results in the results grid on Designer.
Why was this utility created?
AMP Engine can deliver great performance gains to many users based on their use case. The gains become more compelling when using larger data sets and more powerful underlying hardware. However, since this engine was architecturally designed to leverage multi-threaded processing, it produces a different order of output records in some cases. Subsequently, ‘Engine compatibility mode’ was introduced in AMP Engine to produce output record order as close to the original engine as possible. As users embark on a path to converting their original Engine workflows to AMP workflows, Alteryx created the Engines Parity Utility to provide users insights into execution and performance differences. This will allow users to be better informed and to more confidently make the leap to AMP and the benefits it offers.
Where can I find it?
You can download the tool from Alteryx Community Gallery.
Link: Engines Parity Utility
Is there any special requirement to run this tool/app?
Engines Parity Utility is available starting from the 2022.3 release version. To use the tools available in the utility, users need to install at least Designer version 2022.3.
What are all the available configurations in the tool/app?
Within the tool configurations, you will find the following configuration settings.
- Browse to the directory containing workflows you want to run using this tool or specify its path. Note: The tool will navigate through all subdirectories.
- Specify an output location for results. By default, it is saved to your defined temp location.
- ‘Run AMP Engine with compatibility mode’ is checked by default to allow users to get maximum matched results.
- 'Filter only workflows built with original engine' checkbox can be used to only pass workflows that were built with the original engine through the tool.
- 'Global record limit for all inputs' can be used to limit the number of records passed for each workflow comparison, thus avoiding long run time for utility and overwhelming the system in case any workflow points to a very large dataset.
What all information does the output provide?
Users can interact with the two output anchors on the tool to look at matched and mismatched workflow within the result grid.
The tool/app produces an Excel file output report that contains a record for each workflow in the directory and provides details such as:
- Location of each workflow in the directory
- If the tool was run using compatibility mode
- If the run was a success or failure with both E1 and AMP respectively
- If the resulting output was a match or mismatch
- The time taken for workflow execution by E1 and AMP engine respectively
- The performance difference between the two Engines for the given workflow
- List of tool IDs that produced mismatched results between the two engines
Alteryx is confident that AMP will keep you ahead of what your growing data demands. Now, you can be more assured about the AMP journey with the Engines Parity Utility. Upgrade to our latest 2022.3 Designer version and download Engines Parity Utility from our Community Gallery page to gain insights into the power of AMP Engine. For any questions, comments, or feedback, reach out to us with the comment box on the download page or contact us via email at MDG_AMP_PM@alteryx.com.
