Community Spring Cleaning week is here! Join your fellow Maveryx in digging through your old posts and marking comments on them as solved. Learn more here!

Engine Works

Under the hood of Alteryx: tips, tricks and how-tos.
DavidHa
Alteryx
Alteryx

Alteryx Architectures_Banner_999x240-02.png


Alteryx Architectures - Introduction

Alteryx Architectures - Starter Architectures

Alteryx Architectures - SAML SSO Authentication

Alteryx Architectures - Workload Management (you are here)

Alteryx Architectures - Resiliency and High Availability

Alteryx Architectures - Alteryx Server Demo Environment


Welcome to the next installment in the  Alteryx Architectures blog series.  In this edition, we’ll take a look at various workload management techniques from an architecture perspective that can help ensure Alteryx Server environments are able to process incoming workloads efficiently.  As mentioned in previous blogs, we always recommend working with your sales representative through a sizing exercise to ensure the Alteryx Server environment is large enough to support the user and job execution workload it is expected to support.  However, there are certain techniques beyond environment size to help ensure workloads are managed appropriately, and that is the focus of this blog. 

 

QoS Prioritization

 

Quality of Service (QoS) prioritization allows an Administrator to reserve specific Worker nodes for certain workloads based on priority levels.  The Quality of Service options are defined in the Worker System Settings.   

 

QoS Setting Description Allowed Jobs by QoS

0

All jobs are allowed 0 - 6
1 Medium Priority and above 1 - 6
2 High Priority and above 2 - 6
3 Critical Priority and above 3 - 6
4 Chained Apps and above 4 - 6
6 Workflow Validation 6 only

 

 
A couple of examples.  If an available Worker has a Quality of Service of 0, the Worker handles any request. But a Worker with a Quality of Service of 3 only handles jobs that have a priority of 3 or higher. This allows you to reserve resources for higher-priority requests. 

 

With an understanding of how this feature works out of the way, a couple of examples where customers have found this feature highly beneficial. 

  • When an organization has business critical workflows that need to run immediately and not sit in the job queue, having a Worker node with QoS = 2 (equates to High priority) and scheduling these jobs with the High Priority setting will ensure these workflows are able to execute on the dedicated Worker. 

  • If the Alteryx Server environment consists of a Controller only node, or a Controller + Gallery node, it can be helpful to also configure this node as a Worker with QoS = 6 (Workflow validation only).  This allows the node to process Workflow Validation jobs only.  These jobs are usually very quick and light weight as no data is actually processed. This is helpful because we don’t want users publishing Workflows from Designer to have to wait for their workflow validation request to sit in the queue for a chance to run.  Such a setup allows these validation requests to happen quickly, and not impact the more important jobs running on the real Workers.  

 

An example environment which has a Controller servicing Workflow Validation requests, one High Priority dedicated Worker, and two regular Workers is shown below.  

 

DavidHa_1-1644433145878.png

 

Worker Tagging / Job Assignment 

 

The Job Assignment setting allows Workers to be assigned to run specific jobs.  This technique is more commonly referred to as “Worker Tagging”.  The process is simple, in the Worker System Settings add a job tag (or tags) to the Worker.  Then when submitting or scheduling a job, simply specify the job tag to use which will direct job execution to the desired Worker.  Note, the ability to specify a job tag is a user level permission that must be enabled before a user can assign jobs to Workers.  A couple of best practices to consider when working with Worker Tags:

 

  1. Ideally each Worker Tag should be associated with 2+ Workers for redundancy.
  2. At least one Worker should be configured to run "untagged" jobs.

 

With a brief understanding out of the way, let’s now review several techniques for leveraging this from an architecture perspective that many customers have found helpful. 

 

Worker Tagging by Data Location 

For global organizations with data sources dispersed across multiple regions, a strategy that can be helpful is to place Alteryx Server Worker nodes near data sources, and leverage Worker Tagging to ensure jobs interacting with these data sources are executed on the Worker closest to the data.  This allows for more efficient job execution as latency of data movement will be minimized, and in the case of cloud deployments, could save on costs as data doesn't leave the region (vpc).  An example architecture with three Worker nodes in different regions is shown below: 

  

DavidHa_2-1644433145879.png

 

Worker Tagging by Department 

A very common use case of Worker Tagging is for organizations which are using Alteryx Server to support multiple departments, and they would like isolation of job execution between departments.  In this scenario Worker Tags can help ensure that data processing only occurs on the respective department’s Worker.  This can be useful when different departments require different security settings, have different capacities, or in support of departmental charge-back models. 

 

DavidHa_3-1644433145880.png

 

Worker Tagging by Execution Type (scheduled vs on demand) 

Possibly the most popular use case of Worker Tagging involves having separate Workers for scheduled jobs vs on demand / ad hoc execution.  Often, there’s a different job profile for scheduled vs ad-hoc executions.  Scheduled jobs can typically involve larger data sets and take longer to execute.  Users running ad-hoc executions don’t want to wait for their job to sit in the queue for a chance to execute.  Another scenario where this could be helpful is with data source access.  Perhaps only the Scheduled Worker is setup to access production data sources while the ad-hoc worker is setup to access a dev or test instance of the data source. 
 

DavidHa_4-1644433145880.png

 

Worker Tagging by Workflow Function 

The final technique we will cover in this blog is using Worker Tagging based on the workflow function/pattern.  Perhaps Alteryx Server is being used to perform processing ranging from Prep & Blend type scenarios, Spatial processing, enrichment with data insights, and even building predictive models.  These all have different workflow patterns, resource usage, and in some cases additional library requirements.  Separating these out by Worker can prove beneficial in certain situations to help isolate various processes to specific Workers.  This can also be helpful from a resource utilization / capacity standpoint.  Jobs that are building predictive models can have considerably higher resource requirements and may perform better on a larger server machine.   

DavidHa_5-1644433145881.png

 

Summary 

 

In this blog we have covered two techniques for managing workloads on Alteryx Server, QoS Prioritization and Worker Tagging.  Each of these capabilities offer several benefits to ensure workloads are handled efficiently and high priority jobs are allowed to execute as quickly as possible.  If you have any additional topics you would like to see discussed in this series, please leave a comment below.  Thanks!

David Hare
Senior Manager, Solutions Architecture

David has the privilege to lead the Alteryx Solutions Architecture team helping customers understand the Alteryx platform, how it integrates with their existing IT infrastructure and technology stack, and how Alteryx can provide high performance and advanced analytics. He's passionate about learning new technologies and recognizing how they can be leveraged to solve organizations' business problems.

David has the privilege to lead the Alteryx Solutions Architecture team helping customers understand the Alteryx platform, how it integrates with their existing IT infrastructure and technology stack, and how Alteryx can provide high performance and advanced analytics. He's passionate about learning new technologies and recognizing how they can be leveraged to solve organizations' business problems.

Comments
Brinker
8 - Asteroid

Great article thank you so much, @DavidHa. I was actually looking for excatly this for a couple of weeks now.
Looking forward to the next post in the series!

alberto_herni
9 - Comet

Nice article @DavidHa, I never considered to use the tag worker for execution type (scheduled vs manual), that's really interesting idea!