Alteryx Service: Workflow queue round-robin processing

[_enrique_javier]

Dear Community,

We have 3 servers where the Alteryx Service is setup as follows: server 1 is the Controller, servers 2 and 3 are Workers only; workers can run more than one workflow simultaneously, and when adding workflows to the queue, server 2 always picks them up first up to its limit, then server 3 kicks in; so, simple question: how can we make workflows to run in a round-robin fashion to better use our resources?

Thanks,

Enrique Javier

[KevinP]

@_enrique_javier The controller doesn't have any settings or capabilities to ensure workflows are assigned in any particular way. A common misconception is that the controller assigns queued jobs to the workers based on available work slots. However, this isn't the case as the controller doesn't keep track of workers at all. Instead the controller only manages a queue of jobs and assigns those jobs to a worker at the workers request. The workers meanwhile will consistently reach out to the controller to request a job from the queue if they have an available work slot. If there are no jobs currently in queue the controller just tells the worker to check back later. If there is a job in queue that job is assigned to the worker that made the request, and if the worker still has free work slots it will continue to request jobs periodically until all slots are filled. As such jobs are assigned in a first come first serve type of manner and there is no guarantee any one job will go to a particular worker. The advantage to this is that workers can come and go without impacting the controller at all, and the controller will not attempt to assign jobs to a worker that isn't available. 

[_enrique_javier]

@KevinP Thanks for your response; I get your answer is: it is what it is; however, workload balancing is a very common use case, is there a way to implement any kind of workload balancing that you know of? Can i use the System SettingsàWorker ConfigurationàQuality of Service setting to balance the load somehow? Care to explain what the purpose/actual mechanics of setting the QoS values to something different from 0 (zero) are?

[KevinP]

@_enrique_javier The QoS setting for the worker is used to limit a worker to only execute certain types of jobs. This is useful most often in large high use environments where you may want to reserve a worker for specific tasks. For example reserving a worker for workflow validations could prevent users from having to wait an extended period for a worker resource to become available when uploading a workflow. You can find further details on this setting including possible values in our online Server Help documentation.

https://help.alteryx.com/server/current/admin/index.htm#Configuration/SystemSettings/Worker.htm

[_enrique_javier]

@KevinP I read that section while configuring our servers, but it is not clear to me how to assign the QoS value, it is not an option while scheduling the workflow. I tried this: i configured both of our workers with QoS=1, then i scheduled some workflows in the controller, and they were never picked up, until i set the workers' QoS=0. So, as far as my original topic goes, i guess i have come to a dead end here. Thank you very much for your help though.

[KevinP]

@_enrique_javier You can't assign a QoS value to a job manually. These values are determined by the controller based on the type of job. Most workflows will get set with a QoS value of 0. This would include any scheduled workflows, on demand workflow runs from Gallery, and workflows run via the Gallery API. As you noted if you set both workers to use a QoS priority of 1 these jobs will just sit in the queue indefinitely because the worker will only accept jobs with a QoS value at or above their set QoS priority. As noted in the help there are currently 3 QoS values that can get assigned to a job currently. They are:

• Normal workflow execution: QOS = 0
• Chained application execution: QOS = 4 (all apps in the chain aside from the last)
• Workflow validation requests: QOS = 6

With this in mind if you set a worker to a QoS priority of 4 it will only execute chained applications (excluding the last app in the chain as the last app will have a QoS value of 0), and validation requests. Normal workflows will be ignored. If you set the workers priority to 6 it will only execute workflow validation requests (this is the validation that occurs when you upload a file to Gallery).

[Inactive User]

@KevinP If I run the Gallery API from another workflow, say to execute a job or to download a workflow, would this be considered Priority 6 (validation)?

[KevinP]

@Inactive User No, running/queuing a job from the Gallery API would qualify as QOS = 0 for a normal run. The only way I am aware of to trigger a validation job is when uploading a workflow or app from Designer to Gallery. In regarding to retrieving status of a job or its output these API calls do not trigger a workflow run at all so the QOS settings wouldn't apply. 

Something else to keep in mind is that with relatively recent versions of Server (2018.3+) we have add a job prioritization and worker assignment system in addition to the QOS settings I mentioned on my previous post. The priority value can be set in your API call and effects the actual priority of the workflow itself. This is very different from the QOS setting which is set per worker, and tells a worker what workflows it can and can't run. For more information on these options please see the interactive API documentation, release notes, and help documentation for details.

One last note the interactive API documentation currently incorrectly list the expected value for the priority option as a string, but the API expects an integer value instead and will error if a string value is provided. The integer to string value correlations should be as follows:

0 = default

1 = low

2 = medium

3 = high

4 = critical