06-21-2019 12:34 PM - edited 07-15-2021 10:12 AM
This is the first in a series of articles to explore the Alteryx Server System Settings in depth to gain a deeper knowledge of what these settings are used for, and to provide a bit more context to help you determine the appropriate settings for your environment. Every organization's deployment, use cases, and business requirements are different and thus there is no single configuration to fit all. Having an understanding of the implications of each setting is vital in setting up your Alteryx Server for success.
We’ll explore the system settings for Alteryx Server as of 2019.2, through a series of articles for each component, starting with the Engine.The Alteryx Engine consumes Alteryx workflows and provides high-speed data processing and analytics functionality. This process can be entirely self-contained in Alteryx Designer, scaled across an organization by the Alteryx Service, or deployed in the cloud by the Alteryx Gallery.
The Alteryx Server System Settings configuration wizard allows end users to modify the Engine configuration settings:
Any changes to the configuration settings in the System Settings wizard are applied to the RuntimeSettings.xml file, found at:
%ProgramData%\Alteryx\RuntimeSettings.xml
Note, this file should never be modified through an editor unless specifically requested from Alteryx Customer Support.
A few of these settings are self-explanatory and sufficiently described in the Help. However, many of these can often be confusing, and some have performance implications that aren’t well understood. The subsequent sections will dive into these settings in more detail.
“The Engine Temporary Directory is the place where temporary files used in processed workflows and apps will be placed. This setting should point to a location that is safe to write large amounts of files.”
The Engine Temporary directory is a parent directory in which each workflow executed creates a sub directory for storing data needed during the processing of the workflow. Below is not an exhaustive list, but includes the most common types of data stored in the Engine temporary directory:
These files are all deleted when the workflow processing is complete. For a Designer user, this means when the Workflow is closed or upon initiating another execution of the Alteryx Workflow in Designer. For a Server user, this means upon completion of the workflow execution. Note, the Engine Temporary Directory is used when running workflows from Alteryx Designer. When running on an Alteryx Server, the Worker’s “Workspace” directory is used to store these temporary files.
Recommendations
“Define the maximum amount of memory to use to consume data for each output anchor for tools in a workflow. Thedefaultvalue typically does not need to be changed.”
This setting applies when running in Designer only as mentioned in the Browse Everywhere section. The Browse Everywhere feature allows a user to see a sample of the data (up to Memory Limit Per Anchor in size) at the output anchor of every tool. The default value is 1024 KB (1MB).
This is a great way to analyze your data without using the Browse tools which will write the entire dataset contents to disk.
In the event of the Temporary Directory not having enough space to create the .yxbe file, the following will be logged:
Warning - Alteryx: Disk space on temp drive running low. No browse everywhere data created.
Recommendations
"This is the minimum amount of memory that the Engine will consume while performing operations such as Sorts and Joins within a workflow or app. Generally, this value should not be changed.”
The Engine "Default sort/join memory usage" setting and the Worker “Maximum sort/join memory usage” setting have generated a lot of questions that I hope this section clears up. There are three key points to clarify:
If the Sort/Join memory setting requested exceeds the amount of physical RAM available, Alteryx will revert to a lower value that is safe to commit. In that case a message like the following will be logged in the Alteryx Engine logs:
00:00:0.003 - Alteryx: Allocating requested dedicated sort/join memory would be more than available physical memory. Reverting to 2912MB of memory.
Recommendations
“Define thenumberofprocessingthreadstools or operations can use. Thedefaultvalue is thenumberof available processor cores plus one. Generally, this value should not be changed.”
This setting is determining the number of processing threads that multi-threaded tools can use. The multi-threaded tools are identifiable in thePeriodic Table of Alteryx Tools. (Sort and some Spatial tools highlight the list).Configuring a higher value here may facilitate more parallelism which may result in faster completion times in the execution of these multi-threaded tools, assuming the machine has the capacity to use the specified number of threads. Specifying a higher than needed thread pool size can lead to an over-committed system in which the CPU is constantly context switching between threads which may produce longer processing times.
In Windows Task Manager, the "Logical processors" metric shows us the maximum number of concurrent processing threads on that server:
Recommendations
“Select if you are running other memory intensive applications simultaneously. It is also recommended that this setting be checked for a machine configured to run the Gallery.”
The Windows Scheduling Priorities doc is a great resource to understand why this setting is important. To summarize, Windows assigns time slices of CPU time to processes based on their priority level. Applications with a higher priority will be given more CPU time than applications with a lower priority. This ensures higher priority applications get more processor time when the system is heavily utilized.
Most applications default to “Normal” priority. Some critical Windows processes have a “High” priority, such as the logon and desktop window manager processes. Alteryx installs all Server components (AlteryxService, Gallery, Designer, etc…) with the “Normal” priority. By default, this includes the Engine as well. This could lead to a scenario where resource intensive workflows are running, and the Alteryx Service layer, the Gallery, or even Designer are struggling to get CPU time since they all share the same priority level. This also could inhibit mouse & keyboard inputs, and background disk flushing.
Running the Engine at a lower priority will allow these components to remain responsive even during periods of heavy workload processing.
Recommendations
The Alteryx Engine has many settings that can impact workflow performance and resource consumption. Understanding how each of these settings are used by the Engine, and how some settings even impact others, will allow administrators to configure Alteryx Server optimally for their environment and usage.
Hi David,
I found this post very useful. Written in a very lucid manner. Everything made sense and i am happy that i can use these recommendations and settings at my workplace.
My only doubt is why would i see the files ( Engine_*.yxdb and AYXAsset_*.xml) under Engine folder dating back from Nov 2019.
You mentioned these files are all deleted when the workflow processing is complete. Iam currently using Alteryx Designer directly on the machine where Alteryx server is installed. Did i missed anything or misinterpreted? Thanks ahead for your response.
Thanks,
Varun
This was interesting!
My experience differs some from the detail outlined above. I don't consistently see that
"These files are all deleted when the workflow processing is complete. For a Designer user, this means when the Workflow is closed or upon initiating another execution of the Alteryx Workflow in Designer. For a Server user, this means upon completion of the workflow execution."
Instead, I commonly see that the temporary files are purged when I exit Designer or stop the AlteryxService. If my computer crashes or I force stop the AlteryxService (because I'm impatient about "Stopping" status, for example), then the files don't get purged. Your mileage may vary.
Hi @DavidHa ,
This post is very important to see if your environment is underutilized. And all the information you've brought to us gives us more flexibility to get the performance that you want.
Thank you very much for share this knowledge for all.
This was immediately helpful, thank you!