Alteryx AMP (Bug 🐛 or Defect 💔): Limit Records by Dynamic Input

[MarqueeCrew]

Hi @TonyaS and @JarrodT  and AYXCommunity,

I am creating a new workflow using #AMP and encountered an issue.  The template for my dynamic input sets a record limit (e.g. 2,000,000 records), but Alteryx doesn't honor that record limit in the execution of the workflow.  All 258,838,091 records are passed through the dynamic input tool downstream.  This doesn't just take longer (potential definition of the term: defect), but it puts the wrong (definition of bug) count of records into the system.  Here's a picture of the workflow (with counts):

Once I find the most recent file, I read the file using the dynamic input tool.  Without AMP, this runs in roughly 13 seconds.  With AMP it runs in 52:50 minutes (3,170 seconds).  Here is the log:

Start: Designer x64: Started running at 06/28/2022 9:54:35
Info: Designer x64: The Designer x64 reported: Running at a Low Priority.
Info: Designer x64: The Designer x64 reported: This is AMP Engine; running 32 worker threads; memory limit 15.1 GB.
Info: Directory (1): 3 records were generated
Info: Designer x64: The Designer x64 reported: Beginning to compact waiting packets to reduce memory usage
Info: Designer x64: The Designer x64 reported: Beginning to swap waiting packets to temp files to reduce memory usage
File_Input: Dynamic Input (7): T:\_DATA MAINTENANCE\Acxiom Build\MAIL FILES\MQC_JUL22_FINAL.txt|20000 records were read from "T:\_DATA MAINTENANCE\Acxiom Build\MAIL FILES\MQC_JUL22_FINAL.txt" (limit reached)
Info: Dynamic Input (7): 258838091 records were read from 1 files/queries
Info: Append Fields (10): 258838091 target records generated 258838091 output records
File_Output: Browse (2): E:\Temp\Engine_38316_5b8955b8bd781445a8a2967152e2e7a9_\Engine_38316_ed29b5f60e09a24fa9d236060fe5cb4f~.yxdb|258,838,091 records
End: Designer x64: Finished running in 52:50 minutes using AMP engine.

A potential work-around is to turn Amp OFF 📴 and insert a SELECT tool after the dynamic input tool.  Next use CACHE to run at the newly inserted SELECT tool (13 seconds).  Now you can turn Amp ON 🔛 and continue working.

Notes:  Because the SELECT RECORDS macro is not as performant as SAMPLE tool, I am replacing that in my workflow next.  The performance hit was exacerbated by AMP passing 200+ million records into the macro. 

Please consider updating your "known defects" and/or "AMP Differences" documentation to aid others users until a "fix" can be implemented.

Cheers,

Mark

[TonyaS]

Thanks @MarqueeCrew 

I've been chatting with Jarrod about this and this is not a Defect. 

We only supported a list of tools with the first phase of AMP support of Global Record Limit.  We will have another feature on our roadmap to add the global functionality that enables the Record Limit to the Dynamic Input and other tools in the future. I will make sure it gets some attention in the near future for consideration. 

I will also make sure that the list of supported tools is clearly documented in our AMP documentation, so thank you for bringing this to our attention. 

[MarqueeCrew]

@TonyaS ,

With the default to AMP turned on for new workflow, it took 240 times longer to run than it would have taken without amp. I appreciate that you clarified that this is not a defect. This is just what to expect when you use an unsupported tool in a new workflow. 

Cheers,

Mark

[TonyaS]

Well, the missing Record Limit functionality with Dynamic Input in AMP is as it was designed, for now.

It might be worth it for you to go through Support to log a defect about the slowness as something that could be reported by other customers and have cases added by Support, which would result in it being escalated as well as let other customers experiencing this know that it is a known issue to add their own case to.  AMP taking 240 times as long seems unreasonable to me. 

[Aguisande · Accepted solution]

@TonyaS 

I might understand your answer from a software developer PoV, but if we see it from an end user or UX PoV, this is NOT WORKING and needs to be fixed.

Users expect that Tools (ergo workflows) work as expected when they drop tools into the canvas. This is not the situation.

We need to start evaluating this reports from a broader perspective, which is the end user's one and their experience using the tools.

Just as curious as I am, I'd ask myself: "Why is this working perfectly in the Input Data tool?", "Input Data Tool and Dynamic Input tool are using different loading source code that makes them behave different?"

[TonyaS]

I agree that it should work, whether using AMP natively or via the shim back to original Engine if not supported with AMP yet. 

Minimally, we will get the list of AMP tools that support Record Limit documented under the Input section on the differences page here: https://help.alteryx.com/20221/designer/alteryx-engine-and-amp-main-differences  (and the list that do not). 

Even looking back at the feature that was delivered in 21.4, unsupported AMP tools should honor the setting by falling back to e1. So it may well be considered a defect that even that is not happening. 

[TonyaS]

Hi all, 

An update on this is that we have enabled this functionality with AMP for the 2022.3 Release.  The tool level Record Limit in the Dynamic Input tool will work for both e1 and AMP moving forward. 

When the mode - ReadListOfDataSources is selected (doesn’t matter which option of 4 is selected - Append/Prepend/ChangeFilePath/ChangeFileName) the DynamicInput will limit each source. So, for instance we have 3 files in input, limit will be applied for each of them. (For instance limit is 3, there will be 9 records).

When the mode ModifySQLQuery is selected the data will be firstly filtered and only after that the limit will be applied. So firstly all records that match the filter will be selected from the file, and after that the limit will be applied for them.

[TonyaS]

Hello!

More good news, AMP support for the tool level Record Limit in the Dynamic Input tool is also included in the 2022.1.1 Patch 2, which has been released today and can be found at licenses.alteryx.com.