Scheduler Improvement Requests

[TZChandler]

Alteryx Designer is an amazing data tool but it’s partner, "the Scheduler" needs some much needed upgrades. The Scheduler interface that pops up from Alteryx Designer does need a complete make over. I’m not going to address this but rather focus on the functionality that if delivered makes the Scheduler much more useful. 

Today I’m reading our Mongo db scheduler data using an Alteryx workflow and Tableau to show what’s happening on the Scheduler. This dashboard is what we refer to frequently to see the health of our companies data pipeline.  I’ll share both files soon. 

Here are the top 5 features for the Scheduler.

1. Workflow priority ranking. When two or more workflows are scheduled to run at the same time, ‘priority’ sets the order of execution. Priority is set at the time of scheduling the workflow. Values range from 1 to 100. If a priority is not set the default is 50. The ‘priority’ field can be read from Mongo db. 

Why: At midnight we set off several workflows. We want to centrally manage which runs first based on a common ‘priority’ field. 

2. Restrict which controller and workers a specific workflow can run on. Default is to run on all servers. At the time of scheduling a workflow can designated to restrict which server to execute on. This creates a field called ‘restrict’ which server it cannot execute on. 

Why: Some workflows only run on the main controller due to file system references. Also a worker can be tuned for CPU or Disk I/O and workflows that can benefit from this tuning.  Selecting a disk I/O intensive workflow to run on a server tuned for Disk I/O would speed up our workflows. 

3. Set sequence of successfully completed workflows. 

FYI: We used the Runner tool for a short time to resolve this issue but learned quickly that the Runner tool is like a bull in a china shop and brought our server down. The runner tool as it is today is not an option for production work.

Why: This would allow you to run several workflows one after another. For example the first would read from a data source, the second would do calculations on the data and the third workflow would publish the data. All workflows are given a ‘workflow-number’ which can be seen in the scheduler list and read from Mongo db. 

4. If a workflow fails you can set the number of attempts to run successfully. Report attempts greater than 1 in a new field called ‘attempts’ that can be read from the Mongo db. 

Why: Some workflows fail and if attempted to run again may work. This includes issues with locked files and workflows dependent on processes outside of Alteryx. 

5. If a workflow runs more than X minutes the scheduler kills the workflow and reports a workflow error with a unique code called ‘execution-limit’.  This code can be read in the mongo db. The defaut is 90 minutes and can be set to any number of minutes. Each workflow can have it’s own limit. 

Why: Some workflows start to hog resources and need to be killed. If a new workflow is added this is a good way to protect the overall scheduled workflows. 

[JulieM]

Thanks so much for the detailed post @TZChandler.  Prioritization and the ability to assign jobs to specific workers are items that are higher up in priority as far as enhancements go to the scheduling functionality for Server.  I would love to see or be able to understand the information you're pulling out of Mongo around scheduling!  Feel free to let me know if you would be comfortable with getting on a call to talk through this together, or, if you're comfortable with just sharing on this post that's great as well.  Thanks again for the great feedback!

[joeGade]

Ditto ...as @TZChandler

"Today I’m reading our Mongo db scheduler data using an Alteryx workflow and Tableau to show what’s happening on the Scheduler. This dashboard is what we refer to frequently to see the health of our companies data pipeline. "

my use case:

• Definitely needs Prioritization of Jobs, Our uncertified data analysts create's the production jobs that are not tested and puts junk in Prod which causes delays to Certified Jobs. 
• There is no visibility to schedules in Alteryx server, cant kill long running job's, all the jobs go in to queue.
• Overall Poor architecture on server scheduling .... please fix it asap. 

[fharper]

 agreed

[TZChandler]

I’d be glad to share the dashboard and the Alteryx workflow that allows Marketo to monitor the health of our environment. 

I’m at Insprire 2018. Feel free to email me and set something up. 

[JulieM]

@TZChandler great thanks so much.  I will reach out to you directly so we can connect at Inspire!

[adrianloong]

+1

[The_Dev_Kev_Env]

Great Post @TZChandler  ! I was planning on consolidating my schedule ideas, but you've nailed a few of these and already got the discussion rolling.

Some other ideas that were on my list are:

1. To build off Tim's 3rd point, it would be very beneficial to be able to build a scheduling flow within the gallery. Workflow A and B run in parallel and C runs only if both succeed (maybe workflow D runs on a failure). The idea is to start moving the "events" to a gallery feature and make it a clean user-interface to control
2. Create schedules on apps and provide users the ability to input custom parameters for that specific schedule. There was an idea on this earlier (link to earlier idea )
3. Creating public schedules. Currently, if a colleague and myself want to use the results from a scheduled app, we would have to create 2 different schedules (assuming we are downloading the output from gallery). More broadly, I'd like to be able to control who sees the results of an application

Best,

devKev

[mlittletn]

Add to the list the ability to limit the hours/days a workflow runs. e.g. workflow scheduled to run every 10 minutes; but only M-F between 8a-5p.

[KylieF]

Thank you for your idea! Your idea is very interesting, but we have determined that at this time the idea is not something we are planning to do. If at some point we feel that this idea is something we are interested in creating we will change the status “Under Review”.