Memory Limit Exceeded - How to Isolate Cause within Workflow

[amiller2]

Hello -

I have been dealing with memory issues in Designer 2022.3 and the typical solutions are not making any impact on the error.  I have tried overriding the memory usage.  I have tried taking out certain pieces of the model to limit the amount of work it has to perform on a single run.  These changes don't even slightly improve the issue let alone resolve it.

At this point, I believe my best option is to isolate the exact location / tool combination that caused the memory to max out.  Please advise the best way to troubleshoot by isolating SPECIFIC tools within the workflow.  I understand that joins and sorts require larger memory, but I need to be able to locate the specific joins that caused the model to stop importing data from input tools.

My models have dozens of join tools. This model had dozens before it maxed out its memory, so I am having trouble tweaking my model because I do not know which joins are most problematic because joins are key to my model's objective.

Please please provide any insight.  The bulk of the model was built and running just fine and suddenly stopped.

Let me know if you have questions or need specific facts.  (This is my first discussion post so forgive me if I failed to provide enough context.)

[harsh_alang]

Hello,

I would like to understand more, is there some way for you share more about your workflow? Screenshot or may be the yxmd if there is nothing sensitive. When you say model, do you mean workflow or is it an actual machine learning model?

Join tools cache all data at their input anchors before processing the data, are all your join tools being invoked at same time or some are downstream? 

[RobertOdera]

Hi, @amiller2 

1. For your workflow, you have already tweaked memory (the first highlight). Now also Enable Performance Profiling for the current workflow to identify which tools/ places in the flow are taking the most time/ memory (the second highlight).

2. Please share a screenshot of your workflow so that we can help you identify situations where you can:

• use different tools to hog less memory, e.g., Sample Tool versus Unique Tool
• the Select Tool to reduce the number of fields, e.g., Fields that are no longer needed past a certain point in the flow
• inspect all Join Tools to make sure you do not have duplicate fields or differently named fields with the same data values 
• the Auto Field Tool to optimize the field type size
• order of operations in your workflow logic so that you are "slimming" the processed data load as it moves forward without picking up "bloat."
• leveraging Alteryx database, if you can, as .yxdb has an impressive compression
• any other opportunity that might present only after we see what your flow looks like

I hope you find this helpful.

I look forward to positively impacting you once we have the screenshot!

Cheers!

[gautiergodard]

Agreed with @RobertOdera 's solution. Enabling performance profiling should help you narrow down which tools are consuming large amounts of memory.

After your workflow runs, look in the results window and you'll see the time it takes for each tool to process information and the % of that time compared to overall workflow runtime.