Load Balancing from Controller to workers

[htkonkl]

Right now we can set the limit of processes a worker can run and each worker will ping the controller when a slot is available so that it can assign a new process. However, this is not efficient and you can truly load balance because you don't know how busy that worker acutally is...

We need a way that might be simliar to how we do on DataStage GRID environment where the LSF platform balances that workload across the entire GRID.

[KoryC]

Thanks for the idea post @htkonkl - great feedback and ideas around the load balancing we have in server and how it can be improved. We do have some items on the roadmap for the future around the topics of high availability and configuration. We'll keep this idea in mind as well when it comes to better monitoring and balancing workflow executions across your deployment.

With regards to the load on the server - are you looking at being able to balance based off of metrics such as RAM consumption, CPU utilization, and the like, or are there other metrics or indicators you'd like to see the server respect when picking a worker to execute a job on?

[JulieM]

Marking this post as not currently planned but we will definitely be keeping this in mind as we are thinking about things like jobs prioritization with Server.  Thanks!

[KylieF]

Updating this idea's status back to Not Planned, to be in line with past product updates.