We have extended our Early Bird Tickets for Inspire 2023! Discounted pricing goes until February 24th. Save your spot!

Alteryx Server Discussions

Find answers, ask questions, and share expertise about Alteryx Server.
SOLVED

Resource control on scheduled jobs

SeanAdams
17 - Castor
17 - Castor

Hi all,

 

We're a little concerned about how people publish to our prod scheduler and gallery, in-case one of the jobs is incredibly resource intensive, and takes down the server or starves all other jobs.

  • Is there a way, within the server setup, to limit the priority of certain jobs so that they allow other jobs to go ahead of them?
  • Is there a way to choke the total resource usage so that this automatically forces other jobs to wait or slow down until resources are released?

Many thanks

Sean

6 REPLIES 6
MarqueeCrew
20 - Arcturus
20 - Arcturus

@SeanAdams,

 

Sounds like product ideas to me.  You might also be seeing indications that you are growing and might need to get another server instance.

 

Cheers,

Mark

Alteryx ACE & Top Community Contributor

Chaos reigns within. Repent, reflect and restart. Order shall return.
Please Subscribe to my youTube channel.
SeanAdams
17 - Castor
17 - Castor

:-) Hola Mark, happy Sunday!

 

Am I right that based on your response, Alteryx server doesn't yet have the capabity to do this kind of resource control?

I'm sorry to be asking such a basic question - I would have mocked it up myself on a server at home, but I can't figure out how to get a trial license for the server that I can use at home for learning?

 

I'll give more color for the two resource questions below - and possibly (based on your guidance) get these into the Ideas section:

 

Is there a way, within the server setup, to limit the priority of certain jobs so that they allow other jobs to go ahead of them?

This is like setting the priority on a windows thread to "Below Normal" so that more important jobs can go ahead.   Useful if you have a job that you know is pretty intense (like an analytical job, or a big ETL or dimension conforming job) - you want to let the smaller and quicker jobs overtake you.

 

Is there a way to choke the total resource usage so that this automatically forces other jobs to wait or slow down until resources are released?

This is a similar issue - but even the small and quick jobs may all end up being scheduled by users at the same time (for example - users tend to schedule things at even multiples of the hour, so there's a cluster of jobs that all kick off at 08:00; 09:00; etc).   Does Alteryx server have the ability to queue these up so that even though they are all scheduled to start at 08:00, they only start when there is spare capacity on a worker node?

 

Thanks Mark

Sean

 

MarqueeCrew
20 - Arcturus
20 - Arcturus
As delivered, the scheduler is FIFO. Any jobs scheduled should be scheduled below normal to allow for priority to interactive user needs.

Within a single server instance, priority is FIFO. If you have multiple workers, then you will have more control of workflows. To establish a hierarchy of jobs, dependencies and priorities Alteryx doesn't have the feature set of mature scheduling packages like control-m.
Alteryx ACE & Top Community Contributor

Chaos reigns within. Repent, reflect and restart. Order shall return.
Please Subscribe to my youTube channel.
SeanAdams
17 - Castor
17 - Castor

Thank you Mark - have a good Sunday!

MarqueeCrew
20 - Arcturus
20 - Arcturus
Thanks Sean!

It's a short workweek here for me. I am considering bringing up a server on AWS for demos and such. If I do, I will chat with you about access.

Cheers,
Mark
Alteryx ACE & Top Community Contributor

Chaos reigns within. Repent, reflect and restart. Order shall return.
Please Subscribe to my youTube channel.
JohnBell
8 - Asteroid

@MarqueeCrew...it seems like you're familiar with Control M jobs... I'm not too familiar with them...

 

1) It seems like (as an Admin) I don't have visibility to them since they're not scheduled thru the scheduler

2) The only way I see them is thru the engine log files

3) And I have a case where there's a sequence of workflows that run every 2 hours, but during the time one of our Control M jobs is running those workflows fail due to memory allocation issues.  Even though, the log file for the Control M job says it was completed in 4 minutes or less.

 

Can you shed any light on this?  Is it not a best practice to use Alteryx and Control M jobs together?

 

Any input/additional reading material would be greatly appreciated...

 

Thanks!