Running Simultaneous Workflows Across Multiple Workers On One Machine

[clovegrove]

My environment consists of a Main Controller Node with 2 workers and a Worker Node with 2 workers. My goal is to have each node be able to run 2 workflows at a time, but only one on each worker.

I will take one node to use as an example. Within that node are 2 workers and I have sets of workflows that need to be run in order. I have one set on Worker-1 and another set on Worker-2. I want the node to be able to run 2 workflows at a time, but I don't want either of the workers running 2 workflows at a time.

The problem being, when I set "Workflows allowed to run simultaneously" to 2, there is a chance that one of the workers will run 2 workflows at a time. I would think that if you are allowed to create multiple workers on a node that you would have individual control of them. So instead of the "Workflows allowed to run simultaneously" setting being per machine/node it would be beneficial if that would be "Workflows allowed to run simultaneously on each worker".

Is there a way I'm missing to accomplish this that I am missing?

[fmvizcaino]

Hi @clovegrove ,

In each worker node, you also have the system settings configuration, which allows you to set the number of Workflows allowed to run simultaneously.

The controller on your central node controls the queue and each worker node controls its own availability.

https://knowledge.alteryx.com/index/s/article/Adding-a-Worker-Node-1583461562815

Best,

Fernando Vizcaino

[clovegrove]

Hi @fmvizcaino ,

Thanks for the reply! That doesn't quite answer my question though. What if I have multiple workers on one node? If Worker-1 finishes all of it's workflows early, I don't want Worker-2 to start running 2 workflows at a time. My environment consists of:

Main Controller - Worker-1, Worker-2

Worker Node - Worker-3, Worker-4

So the same would be said for the Worker Node, If Worker-3 finishes it's workflows, I don't want Worker-4 running 2 simultaneous workflows.

This is because we have multiple processes that consist of a number of workflows running in order after one finishes. I'll typically put a process of multiple workflows on one of the Workers. This is why I would like to know if I can control this at the worker level instead of the node level.

[fmvizcaino · Accepted solution]

Hi @clovegrove ,

Thanks for clarifying it. Okay, so the problem is with the definition of a worker.

Your main controller (or your worker node) contains only one worker, which can have as many engines as you set. X engines = X simultaneous workflows running.

Therefore, if you set the number of simultaneous workflows to 1 in your worker node, the node will only run one workflow at a time and that will guarantee no other workflow will be run on that server/node/worker.

https://community.alteryx.com/t5/Engine-Works/Alteryx-Architectures-Introduction/ba-p/848734

Main Controller - Worker-1

Worker Node - Worker-2

Best,

Fernando Vizcaino

[clovegrove]

Thank you for answering this for me @fmvizcaino!

I have a better understanding now of what an actual worker is and how it works. One last thing I have a question about is the engine. When you say X engines = X simultaneous workflows running, what do you mean? I have been looking online for an explanation but I'm not seeing it other than stating what you said. How can you increase the engines, or is that just increasing that Workflows allowed to run simultaneously" setting and it is increasing the number of engines behind the scenes.

Thanks,

Chris Lovegrove

[fmvizcaino]

Hi @clovegrove ,

Yes, the number of engines is the same as the number of simultaneous workflows. Here is more about the engine:

 https://knowledge.alteryx.com/index/s/article/Alteryx-Server-System-Settings-Deep-Dive-Engine-158346...

Best,

Fernando