Workflow hanging in windows command line scheduling and not completing.

[warrencowan]

Hi Guys,

I'm experiening itermittent issues with workflows hanging mid way, and also not closing when they're finsihed, when scheduled in windows task scheduler to run via the command line.

I have no problems running them individually in designer.

Anyone familir with such an issue?

Its a process that runs every 5 minutes, and runs an alteryx workflow looking for a qualifying customer, and if found, will then run a batch macro for them (with lots of batch macro ops within that).

Most runs will come back blank if no qualifying customer is found and shut down, but will run the macro if one is found, marking it as running so it doesnt come through again.

Obviously if one is found then the task will be occupied for some time (30 mins max) , and so the windows task which comes round again 5 minutes later will run another instance in parallel to make sure the next customer isnt waiting.

Is the parralelisation the problem, or could it be something else.

Machine spec and bandwidth is gratuitously overpowered so I dont suspect an issue here, but am gretaful for any pointers.

best

w

[BrandonB]

If a task takes 30 minutes as an example, would your script be picking up that same file every 5 minutes as the workflow runs? If this is the case it sounds like there may be 5 workflows running at the same time at one point in this scenario for the same qualifying customer. 

What is the mechanism you are using to find a qualifying customer? Would it make sense to set this to check every 30 minutes instead? 

[BrandonB]

It could also be the case that by kicking off multiple workflows using the cmd that you are running more workflows than suggested for the machine simultaneously rather than queueing them up via the server api. The recommendation is # of cores your machine has divided by 2. So a 4 core machine can run 2 workflows simultaneously.

[warrencowan]

Hey @BrandonB 

Yes it is running the same workflow. This workflow is like a wrapper for managing the queue and funneling it down to a macro. The workflow can run without picking the same customer twice. I wont bore you with the detail, but it references a list, takes one and marks it off so to speak before running. So next time the workflow fires up the list is minus the one already in play. 

When it does run the main part, its running a macro set with the customers parameters, which will determine everything from logic to read and write locations so the same resources arent being utilised. (Although now you mention it, I wonder if there's an occasional text file of static settings or something embedded in one of the macros that might consitute a single resource that could find itself being requested simultaneously by paralell instances? Worth checking?)

I'm set up to run 6 flows concurrently via advanced settings so maybe that's worth looking at. My machine has 256gb or ram and an AMD 64 core processor so it can take the strain. I could do more.

There are a number of other minor shceduled routines that run, (some through the windows and some through alteyrx scheduler) although they are fairly trivial. Nothing that would hold it up for more than a couple of minutes, where as Im seeing hangs of hours sometimes.

[BrandonB]

When you mention that a workflow is "hanging" that makes me wonder if there is a script maybe you have executing that isn't properly exiting which I could see possibly causing a workflow not to complete. Have you already taken a look at the workflow logs to determine where the workflow is hanging at? 

[warrencowan]

it does vary, which is a real pain. I have a suspicion that the last DB write in the chain is a recurring issue. I've scrolled through the command window while the hung task is running, and it often looks stuck after a db write. The write was successful by all acounts, and quite some time before hand, but the batch macro doing it woudln't complete having finished the run.

We tried moving to a stronger db env just in case but with no resolution.

fyi each customer writes to its own dedicated DB and tables, albeit in same server.

Am using the bulk sql as my dirver for upload.

[BrandonB]

I might suggest explicitly writing the workflow messages to a log file as explained here: https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Save-workflow-messages-log-in-database...

That way you can see exactly what line it is stopping at. You may also want to "Show All Macro Messages"