Hi Community,
we have experienced a strange scenario today on the server.
In brief there were a couple of jobs running manually and were stuck in validation phase since last night.
We have set the limit of killing jobs after 7200 seconds but these because run manually were not detected and hence our hundreds of regular scheduled jobs which kick off at 6AM were queued until we have killed the 2 culprits.
I have seen lots of creative ways to detect such manual jobs if they surpass the limit however would be possible to empty the queue at a specific time (lets say 2AM) so our regular morning jobs are safeguarded?
Thanks
Luca
