Workflow stuck in "Initializing status"

[JohnBell]

Hi,

Every now and then, I see workflows that are stuck in the "initializing status" in the queue.

I can't delete it.  So I'm forced to restart the worker/controller, or in some cases, it fixes itself without any input from me.

Has anyone else experienced this behavior?  What did you do to fix it?

Thanks,

John

[MarqueeCrew]

@JohnBell ,

I can't say that I've ever had this happen.  When you say that you restart, are you actually restarting or are you using TASK MANAGER on the server to stop and start the service?  That could be faster.

I worry that the MongoDB entry might be set that the job is running and you might only be running a limited set of concurrent jobs.  If that is the case, you'll need to cleanup the MongoDB (support@alteryx.com).

Cheers,

Mark

[JohnBell]

I'm actually restarting the Alteryx service.

AlteryxSupport initially told me that it's stuck in "initializing" is due to communication issues with the controller/worker due to over-utilization.

But I'm wondering if this is common?  Because when the "initializing" error occurs the server isn't experiencing much activity compared to other times of the day.

Thanks,

John

[MarqueeCrew]

Concurrent runs includes SAVING.  So if you allow for 2 concurrent jobs and they are both running, then you can not save to gallery (normally).  You can uncheck the "Validate workflow then save" option and save.  But if you validate, then you become a job in the queue.

Cheers,

mark

[sd83893]

We are experiencing similar error "Initialization" Status....

[JohnBell]

Hi,

We have version 2018.2 and we recently converted our Controller to run scheduled workflows; therefore, we now have 2 worker machines that can run 3 workflows concurrently for a total of 6 workflows running concurrently.

AlteryxSupport mentioned that we are over-utilizing our server and that we need to increase the cores, Vcpu's, etc.  (which kind of tics me off because Support & Reps knew our settings, knew we were updating to 2018.2, and knew we were adding the additional worker capacity on the Controller), but didn't bring this up previously.

Sometimes the issue will fix itself without intervention.  In other cases, you need to restart the service on the machine, and then I see the workflow finally runs and completes.  During the "stuck in Initialization status" the workflow will not run, even when it is scheduled to run at a later time.

Please let me know your findings...

Thanks,

John

[JohnBell]

@sd83893 did you get this fixed?  What did you find out...does it agree with my findings?

Thanks,

John

[sd83893]

Hi John,

I should have explained our environment better, we have one controller, three worker, and each worker can execute 8 workflows at any given time, so in total of 24. We get initializing status even when there are no workflows that are queued. We know the challenge is on saturation in our environment, and we are suggesting if some sort of resiliency can be put in place

Thanks

shine

[JohnBell]

@sd83893 

Hi,

It would be interesting what Support recommends to fix this.

If you have 8 workflows per machine (based on 2 cores/per 1 concurrent workflow) you should have 16 cores on each worker.

Currently we have 2 workers (the Controller is enabled as a worker); we are thinking of adding another dedicated worker.  And we're initial thinking of 8 cores per machine (so 4 concurrent workflow on each machine for a total of 12.  Then of course there is the RAM consideration, etc.

Good luck...keep me posted, and I as well will keep you posted.

Thanks,

John

[levell_x_dunn]

Hi @JohnBell ,

Unfortunately we see the same issue.

We have 1 controller and 5 workers so we have plenty of space but now and then Alteryx goes buggy and the workflow gets stuck in initializing and it will stay there for days if we don't restart the service or sometimes the entire server.

Which is a problem for us since we have jobs running 24 hours so there is always at least one job on each server. So we have to fail someones job just to restart the servers.

Alteryx hasn't found a fix for this yet.