Resource control on scheduled jobs

[SeanAdams]

Hi all,

We're a little concerned about how people publish to our prod scheduler and gallery, in-case one of the jobs is incredibly resource intensive, and takes down the server or starves all other jobs.

• Is there a way, within the server setup, to limit the priority of certain jobs so that they allow other jobs to go ahead of them?
• Is there a way to choke the total resource usage so that this automatically forces other jobs to wait or slow down until resources are released?

Many thanks

Sean

[MarqueeCrew]

@SeanAdams,

Sounds like product ideas to me.  You might also be seeing indications that you are growing and might need to get another server instance.

Cheers,

Mark

[SeanAdams]

:-) Hola Mark, happy Sunday!

Am I right that based on your response, Alteryx server doesn't yet have the capabity to do this kind of resource control?

I'm sorry to be asking such a basic question - I would have mocked it up myself on a server at home, but I can't figure out how to get a trial license for the server that I can use at home for learning?

I'll give more color for the two resource questions below - and possibly (based on your guidance) get these into the Ideas section:

Is there a way, within the server setup, to limit the priority of certain jobs so that they allow other jobs to go ahead of them?

This is like setting the priority on a windows thread to "Below Normal" so that more important jobs can go ahead.   Useful if you have a job that you know is pretty intense (like an analytical job, or a big ETL or dimension conforming job) - you want to let the smaller and quicker jobs overtake you.

Is there a way to choke the total resource usage so that this automatically forces other jobs to wait or slow down until resources are released?

This is a similar issue - but even the small and quick jobs may all end up being scheduled by users at the same time (for example - users tend to schedule things at even multiples of the hour, so there's a cluster of jobs that all kick off at 08:00; 09:00; etc).   Does Alteryx server have the ability to queue these up so that even though they are all scheduled to start at 08:00, they only start when there is spare capacity on a worker node?

Thanks Mark

Sean

[MarqueeCrew · Accepted solution]

As delivered, the scheduler is FIFO. Any jobs scheduled should be scheduled below normal to allow for priority to interactive user needs.

Within a single server instance, priority is FIFO. If you have multiple workers, then you will have more control of workflows. To establish a hierarchy of jobs, dependencies and priorities Alteryx doesn't have the feature set of mature scheduling packages like control-m.

[SeanAdams]

Thank you Mark - have a good Sunday!

[MarqueeCrew]

Thanks Sean!

It's a short workweek here for me. I am considering bringing up a server on AWS for demos and such. If I do, I will chat with you about access.

Cheers,
Mark

[JohnBell]

@MarqueeCrew...it seems like you're familiar with Control M jobs... I'm not too familiar with them...

1) It seems like (as an Admin) I don't have visibility to them since they're not scheduled thru the scheduler

2) The only way I see them is thru the engine log files

3) And I have a case where there's a sequence of workflows that run every 2 hours, but during the time one of our Control M jobs is running those workflows fail due to memory allocation issues.  Even though, the log file for the Control M job says it was completed in 4 minutes or less.

Can you shed any light on this?  Is it not a best practice to use Alteryx and Control M jobs together?

Any input/additional reading material would be greatly appreciated...

Thanks!