Alteryx Designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer and Intelligence Suite.

Issues faced when using Alteryx AMP engine

Rihan_Mohamed
7 - Meteor

Below are issues which we faced when working with AMP engine feature. Can anyone help us to resolve the below issue and explain more about AMP feature.

 

  1. Delimiter (\0):- We cannot use AMP engine in cases where there are spills or splits in data and single row gets split to multiple rows. In such cases we import files without splitting on delimiters and later on combine rows to create final table. AMP data is processed in parallel packets which are comprised of rows chosen randomly from data instead of grouping rows sequentially. This leads to distortion of data.
  2. File Format:- AMP reads CSV and YXDB files considerable faster whereas for other data types files is almost the same.
  3. Record Limit:-  There is no provision of test run by limiting records at initial run mostly to test workflow. As AMP overrides the limit and reads whole file.
  4. Data Field Length:- We cannot limit the data length as AMP allocates good memory to each column so there is no room for blockage or truncation of data. Might be a challenge incase it is being used as input to database resulting in more space consumption.
  5. Multiple Joins:-  We have observed that incases where we had multiple joins in the huge data sets then it created a deadlock which resulted in no output resulting in loss of 3-4 hours. This might be due to parallel sharing of resources whereas in normal case it would happen sequentially
  6. Iterative Macros:- It results in unnecessary or even wrong output if iteration is too high. Errors are ignored and processing continues. We don’t have experience on this however its based on my experience from Alteryx Community.
  7. Excel Output : Error while writing in multiple sheets of same excel file as AMP attempts to write multiple outputs simultaneously.
13 REPLIES 13
TonyaS
Alteryx
Alteryx

@Rihan_Mohamed 

 

First, I want to thank you for using the AMP Engine and for providing feedback. 

 

I encourage you to continue to report any use case issues that you find with running workflows with AMP Engine enabled. We worked hard to identify differences from the original Engine as well as provide guidance on how to better optimize workflows to run with AMP.  Some of these differences have already been addressed in our documentation. 

 

2. File Formats: AMP supports yxdb, csv, xlsx and sqlLite so those formats should be faster, all others must fall back to the original engine. 

3. Record Limit: We have a feature on the roadmap to enable Record Limit configuration with AMP.

4. AMP attempts to accommodate for future processing of data. 

5. Deadlocks have been addressed with 21.1, 21.2 and there will be additional edge-case fixes in 21.3 that have been identified with the help of ACEs. 

6. Iterative Macros - differences are addressed in the "Main Differences" documentation. 

7. Excel Output - this can be addressed by placing a Block Until Done tool in the branch with the lowest numbered Input tool id. 

 

I also wanted to provide some helpful links to available documentation about AMP Engine:

 

AMP Engine webinar (32 minutes): https://community.alteryx.com/t5/Analytics/AMP-Engine-Discover-the-Untapped-Performance-of-Your-Anal...

 

https://help.alteryx.com/current/designer/alteryx-amp-engine

https://help.alteryx.com/current/designer/Alteryx-Engine-and-AMP-Main-Differences

https://help.alteryx.com/current/designer/AMP-Memory-Use

https://help.alteryx.com/current/designer/tool-use-amp

 

https://community.alteryx.com/t5/Analytics/Accelerate-Your-Analytic-Processes-with-the-New-AMP-Engin...

https://community.alteryx.com/t5/Engine-Works/AMPlify-your-Workflows/ba-p/617590

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Best-practice-with-the-new-Multi-threa...

AlterEverything Podcast: https://community.alteryx.com/t5/Alter-Everything-Podcast/66-The-Alteryx-AMP-Engine-Explained/ba-p/5...

https://community.alteryx.com/t5/Engine-Works/AMP-Engine-Technical-Deep-Dive-Part-1-Why-AMP/ba-p/570...  

https://community.alteryx.com/t5/Engine-Works/AMP-Engine-Technical-Deep-Dive-Part-2-Key-concepts-of-... 

 

Server:

https://help.alteryx.com/current/server/controller#configuration

https://help.alteryx.com/current/server/worker

https://help.alteryx.com/current/server/engine

Tonya Smith
Sr. Technical Product Manager, Alteryx Engines
TonyaS
Alteryx
Alteryx

@Rihan_Mohamed 

I asked around, and we're not sure what you mean exactly by the first item in your list: 

  1. Delimiter (\0):- We cannot use AMP engine in cases where there are spills or splits in data and single row gets split to multiple rows. In such cases we import files without splitting on delimiters and later on combine rows to create final table. AMP data is processed in parallel packets which are comprised of rows chosen randomly from data instead of grouping rows sequentially. This leads to distortion of data.

 

If you could supply a short workflow that demonstrates the issue (as a yxzp so that the data is included), that would allow me to reproduce it and see the issue. 

Tonya Smith
Sr. Technical Product Manager, Alteryx Engines
GabeHCoud
5 - Atom

@TonyaS 

 

I've faced 2 more issues when using AMP.

 

  • Data Cleansing tool: the tool's output sorts the data in an inexplicable way, almost randomly. I isolated the issue to the "Remove Null rows" option, for some reason when ticking this option the data sorting order is not the same as it originally is. However when disabling the use of AMP engine, the issue is solved. 

 

  • When running a workflow with a Batch Macro, the workflow errors on the macro showing the message Record #1: Tool #79: Parse Error at char(4): Unknown variable "Variable", I've checked this "Variable" field and everything appears to be alright with its data, expressions, field names, etc., in fact when disabling the use of AMP engine the workflows runs correctly.
TonyaS
Alteryx
Alteryx

@GabeHCoud 

For the Data Cleansing tool - we have documented that difference: https://help.alteryx.com/20221/designer/alteryx-engine-and-amp-main-differences
and the order difference can be easily solved by using the "Engine Compatibility Mode" to maintain the record order. 

 

For the Batch Macro issue - please go through Support and get a defect created, with a workflow to reproduce the issue. Then it can be escalated to my team to look into. We do have a few issues related to Batch Macros, some are fixed for the next Release - so if there are additional issues we need to understand what exactly they are in order to address them. 

Tonya Smith
Sr. Technical Product Manager, Alteryx Engines
anttirask
5 - Atom

@TonyaS, just wanted to let you know that having AMP Engine on caused my Dynamic Input with a csv file with tab (\t) as a delimiter to go haywire.

Instead of creating a NULL cell it just filled the cell with the next bit of data it found.

TonyaS
Alteryx
Alteryx

@anttirask 

Is there a chance you have quoted fields in your CSV? 

There are 2 optional additional AMP specific configurations for CSV files: 

 

TonyaS_0-1654200380461.png

 

Tonya Smith
Sr. Technical Product Manager, Alteryx Engines
ChrisTX
14 - Magnetar

@TonyaS we are testing version 2021.4.2.35356 with the AMP engine on.  Two issues are below, will be reported to support today.

 

1) With the AMP engine on, my workflow does not read any data from two Input Data tools that read YXDB files.  With AMP off, the data is read in as expected.   After I significantly reduce the tools in my workflow, to simplify it before sending it to support, I can find a few ways to get the YXDB files read in.... like deleting unrelated containers, or changing other unrelated input tools to read a Text Input tool instead of a text file. 

But in my full workflow, those same changes will not force a read of the YXDB file, so I can't test the AMP engine with my full workflow.

 

2) With the AMP engine on, when I open the workflow I get the two errors in the screenshots below.  I also have another version of the same workflow that does not give the memory error, but still gives the 

Error: Designer x64: The Designer x64 reported: InboundNamedPipe GetOverlappedResult:  The pipe has been ended.

 

ChrisTX_0-1662644311760.jpeg

 

ChrisTX_1-1662644329788.jpeg

 

Notes:

Our issue with "InboundNamedPipe" may have been fixed under defect TDSL-4131.  We'll need to test in version 2022.1.

But in the 2022.1 release notes I don't see any defect mentioned for reading YXDB files.

 

Chris

TonyaS
Alteryx
Alteryx

Hi @ChrisTX 
There are 2 formats of YXDB for original Engine and AMP. 

 

If you are trying to read a YXDB that was written with the original Engine using AMP it may be much slower. 
The 2 formats of YXDB are documented here: https://help.alteryx.com/current/designer/Alteryx-Engine-and-AMP-Main-Differences

Here is more documentation on AMP: 

Help Documentation

https://help.alteryx.com/current/designer/alteryx-amp-engine

https://help.alteryx.com/current/designer/Alteryx-Engine-and-AMP-Main-Differences

https://help.alteryx.com/current/designer/AMP-Memory-Use

https://help.alteryx.com/current/designer/tool-use-amp

https://help.alteryx.com/20221/designer/engine-compatibility-mode

https://help.alteryx.com/20221/designer/performance-profiling-amp-engine

https://help.alteryx.com/20221/server/amp-engine-best-practices

 

•AMP Engine Webinar (32 minutes)

https://community.alteryx.com/t5/Analytics/AMP-Engine-Discover-the-Untapped-Performance-of-Your-Anal...

 

•AlterEverything Podcast

https://community.alteryx.com/t5/Alter-Everything-Podcast/66-The-Alteryx-AMP-Engine-Explained/ba-p/5...

 

•Community Blog posts

https://community.alteryx.com/t5/Analytics/Accelerate-Your-Analytic-Processes-with-the-New-AMP-Engin...

https://community.alteryx.com/t5/Engine-Works/AMPlify-your-Workflows/ba-p/617590

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Best-practice-with-the-new-Multi-threa...

https://community.alteryx.com/t5/Engine-Works/AMP-Engine-Technical-Deep-Dive-Part-1-Why-AMP/ba-p/570...  

https://community.alteryx.com/t5/Engine-Works/AMP-Engine-Technical-Deep-Dive-Part-2-Key-concepts-of-...  

Tonya Smith
Sr. Technical Product Manager, Alteryx Engines
ChrisTX
14 - Magnetar

@TonyaS thanks

 

Using version 2021.4.2.35356 with the AMP engine on, I created a small workflow to read in my existing YXDB file (created with our old version 2020.4 with AMP off).

 

I wrote out a new YXDB file. 

 

Using version 2021.4.2.35356 with the AMP engine on: in a brand new workflow, data is read in from the new YXDB file as expected.

 

But my existing workflow, created under our old version 2020.4 with AMP off, will not read either the original or the new YXDB file.  Even dragging in the YXDB file to create a new Input Data tool doesn't work.  I cannot find any way to get my large workflow to read data from the YXDB file.

 

Chris

Labels