Optimisation tool run time

[dgibson]

Hi, 

I have designed and implemented an optimisation model using the optimisation tool in designer and it is solving successfully. Because the data set is large, with approx 2.5M variables, my model's run time is sitting around the 3 hours mark. Although I can live with 3-hours I'm looking for ways to speed up the solving time to allow more frequent iterations and scenario testing. I have already constructed the model using a sparse (SLAM) matrix format. 

In an attempt to speed up solving time I attempted a switch to the AMP engine and, to my surprise, found that the solving time increased by approximately 1 hour! To make sure it wasn't a fluke I have re-run the model multiple times using both AMP and standard engines and each time the AMP engine is around a full hour slower. In one instance I watched the model running with AMP and found that the steps prior to the optimisation ran significantly faster but on reaching the optimisation tool things grind to a crawl. Interestingly, the optimisation engine shows a '?' status when running with AMP:

Obviously I can continue to use the standard engine for the time being, but I'm confused about why the AMP engine would be significantly slower. At worst I would expect it to be about the same. My questions is, is this behaviour to be expected with the optimisation tool, or could there be something in the way my model is configured that is preventing the AMP engine from delivering a performance benefit ?

Any ideas anyone has are appreciated.

[apathetichell]

Someone who is more knowledgeable than me can chime in - but basically as I see it AMP is for parallel/multi-thread running through different parts of a workflow. Optimization tool is a single tool so you are basically channeling multiple streams through a single tool. If you want better performance - increase your system ram - or run in a ram optimized VM.

[dgibson]

Good info, thanks. Unfortunately I've already maxed out my RAM allocation on my laptop but I will explore whether running an instance on a server will be an option. 

[NeilR]

Not all tools take advantage of the AMP engine. Here is the list of those that do - the Optimization tool is not one of them.

[TonyaS · Accepted solution]

AMP will not always be faster, especially with heavier tools that are not yet supported with AMP. 

I do know that 2022.3 will add AMP support for all of the R Macro tools and the R tool itself, including 11 that are available for download from Gallery. So beginning with the 2022.3 Release you will be able to use Optimization with AMP.  

While it should still "work" by falling back to e1 for unsupported functionality, even if tools are not yet converted to AMP tools - some of them may be slower. 

Here is the current external list of AMP Supported Tools: https://help.alteryx.com/current/designer/tool-use-amp 

And here are some other AMP documentation pages and resources: 

•Help Documentation

https://help.alteryx.com/current/designer/alteryx-amp-engine

https://help.alteryx.com/current/designer/Alteryx-Engine-and-AMP-Main-Differences

https://help.alteryx.com/current/designer/AMP-Memory-Use

https://help.alteryx.com/20221/designer/engine-compatibility-mode

https://help.alteryx.com/20221/designer/performance-profiling-amp-engine

https://help.alteryx.com/20221/server/amp-engine-best-practices

•AMP Engine Webinar (32 minutes)

https://community.alteryx.com/t5/Analytics/AMP-Engine-Discover-the-Untapped-Performance-of-Your-Anal...

•AlterEverything Podcast

https://community.alteryx.com/t5/Alter-Everything-Podcast/66-The-Alteryx-AMP-Engine-Explained/ba-p/5...

•Community Blog posts

https://community.alteryx.com/t5/Analytics/Accelerate-Your-Analytic-Processes-with-the-New-AMP-Engin...

https://community.alteryx.com/t5/Engine-Works/AMPlify-your-Workflows/ba-p/617590

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Best-practice-with-the-new-Multi-threa...

https://community.alteryx.com/t5/Engine-Works/AMP-Engine-Technical-Deep-Dive-Part-1-Why-AMP/ba-p/570...  

https://community.alteryx.com/t5/Engine-Works/AMP-Engine-Technical-Deep-Dive-Part-2-Key-concepts-of-...

[dgibson]

Thanks for the info Tonya, my knowledge of engines is still fairly rudimentary so I'll have a read and see if I can pick up a few things. I'll also keep an eye out for v. 2022.3.