cancel
Showing results for 
Search instead for 
Did you mean: 

Poormans Scheduler

SOLVED
Highlighted
mutuelinvestor
Asteroid

I'm a startup that's not in the position to purchase the Scheduler Add-on yet, so I was curious to know what others have done as an alternative to schedule to schedule scripts.  At the inspire conference a couple of years ago, I spoke to a user who was successfully using windows scheduler.  

 

If you have an approach you are willing to share, I would be very interested to hear about it and the steps required to implement it. 

 

Thanks,

 

Jim

  • Scheduler

@fharper gave a pretty cool presentation to the Tampa User Group last year. I *think* it may be exactly what you're looking for, but I've tagged him to see if he has any insights. 

fharper
Asteroid

Thanks Patrick!  I built a scheduler system leveraging the command line execution feature you get when you buy the scheduler option for a Desktop Alteryx license.  

 

I think the slide show, which is more of an overview, is still available through the Tampa User's Group page in the community.  It was Nov 2016 so probably listed as 4th quarter meeting.

 

It is more sophisticated in its capabilities than what the Alteryx Scheduler offers.  If you want details you can contact me directly.

anthony
Bolide

The super cheap version is to use autohotkey script and create a .exe to open a yxmd file and hit run button or control R it.

 

That you can schedule in windows scheduler.

 

There are desktop automation solutions that do the same thing but can be $1-2k

fharper
Asteroid

you can probably sort my scheduler system out from the powerpoint posted from the Nov 2016 Tampa User's Group.  It is a bit more evolved in some ways than Autohotkey and macroreader.  We initially used macroreader to script our order of execution and automate the keystroke responses to make the workflows or apps run without user intervention.  You had to make everything an app as I recall or it would not run from command line, unless you bought the scheduler option which was not available back then actually.  there were other issues with that tool and approach as well.

 

Long story short we paid the smaller fee for the scheduler option to a designer license when it became available versus the BIG bucks for the Server version that comes with a scheduler.  Even so the Alteryx scheduler was insufficient for our needs, basically a repackaged Windows Task Manager.  But with the scheduler option I can run any workflow from the command line.  so I built a control file for my scheduler criteria, a job to evaluate the logs I write and the schedule criteria I designed and determine what jobs/workflows to run then feed that into another workflow to build batch files to run them.  

 

This is all run from another batch file that is run by the windows task scheduler every 10 minutes.  each of the batch files that run other batch files or jobs writes to log files to record the run and error level for evaluation in the next 10 minute cycle.  if something failed we can automatically re-submit if we set up the criteria to do so.  this is handy as many times we have connection issues with files and DBs that cause failures and will clear up in minutes or some period of time and we can just resubmit until it works if we want, or limit re-submission, by job.

 

I say job because we run VBA and other programs from the scheduler as well, not just workflows.  anything you can run from the command line you can use the schedule system for.

 

I have a calendar feature used to determine if it is a business day, weekend or holiday so we can use it in our schedule criteria.  I have daily weekly monthly and specific day and certain days of week and times of days and specific day or next business day or prior business day logic as well.  stuff you see in a good mainframe scheduler but not in Windows or any other scheduler for under a few thousand a year.

 

A big feature is ability to define predecessors.  Where we have 2 separate jobs that may each extract some data from separate sources and then a third job that merges the data and does final processing and reporting.  We call the first 2 predecessors and can set it up so the last job waits for the first two to finish.  very useful in breaking up big workflows into smaller ones that can run in parallel and then merges in a final flow.

 

The presentation from Nov 2016 gives a good view of that and parallel processing and some other handy tips.

 

If you still have questions or want the code you can contact me.

mutuelinvestor
Asteroid

Thanks for all the great information.  To execute your solution, however, does still require the purchase of the Alteryx desktop license, correct?  If so, this solution may still be beyond what's in the budget this year,  but one I would like to explore in the future.  If you are willing to share files and configuration, I would truly appreciate it.  

 

Thanks again for the great information.  

 

Jim

mutuelinvestor
Asteroid

You had me at super cheap.  I will definitely check out autohotkey.  I suspect this will meet my need just fine until I can go out and purchase the Scheduler Add-on. 

 

Thank you.

fharper
Asteroid

Not sure I caught what you are saying.  I assume you have a desktop license, you can't use Alteryx beyond trial without a license.  

 

There is added cost though for the Scheduler feature added to the desktop Designer so yes...to execute from the command line without intervention or simulation of keystrokes you need to pay the additional cost of the scheduler feature which is several thousand a year I think but your sales guy can answer that.

 

Macro Reader and AutoHot key and a few others out there offer ways to simulate keystrokes to address the manual intervention required otherwise but are not themselves very evolved for scheduling.  A lot depends on how sophisticated your needs are.

 

When we started using Alteryx it was all manual.  then we started using it more and more for cleansing and preparation and extraction because it is so good at it.  this meant we started having more and more things we needed to run repetitively and regularly on simple schedules.  

 

And as things evolved we developed more complex solutions with more varied schedule needs and Windows task manager and macro-reader and such tool were just not meeting the need so we paid the fee to get the scheduler option added to one of our Designer licenses and dedicated that license to running our "production" work and built a scheduler with the variety and flexibility we needed for criteria.

 

I will offer this suggestion though...If you can't pay the several thousand a year for the scheduler option added to a license consider writing a workflow that runs an iterative macro which evaluates the scheduler logic and subsequently runs other workflows.  its sort of convoluted but may be possible.  You can chain one workflow from another without needing manual intervention in the spawned workflow.  Batch macros may also offer opportunities to explore if you take this alternative path.  

 

It is not the path I took so I can offer little more explanation other than the idea being to have a workflow that maybe fires after midnight and essentially runs all day as it keeps iterating on an iterative macro which in turn evaluates logic to fire other macros and flows.

 

On the surface this latter idea sounds straight forward but as always the devil is in the details and I love the scheduler system I already built.

 

I think the product management team at Alteryx is missing a good opportunity with the scheduler feature.  They only started offering it 2 or 3 years ago after enough of us users pushed for it but they did put a hefty price tag on it.  So most places only add it to one or two Designer Licenses rather than 10 or 30 or all.  

 

If they cut the cost to a fourth of the current cost they would probably sell 6 to 10 times as many add-ons netting them more revenue and making more users more engaged and more dependent further penetrating their customer base.  But they don't really promote the feature, I think they fear it takes away from up-selling to the server.  

 

I don't see their scheduler though as a reason to get the server, I see other reasons but not the scheduler.

 

mutuelinvestor
Asteroid

I like your suggestion and will give this a try.   I couldn't agree with you more on Alteryx's approach to Scheduler. I believe they would sell a lot more licenses if they dropped the price os scheduler, but as you said they probably see that as undermining the server sales.  

 

Thanks again for all the good information.  I really appreciate it. 

 

JIm

fharper
Asteroid

One other thing...with cyber security an escalating and evolving battle we found Macro-reader and similar offerings to often be impacted by updates to our software security products and policies.  

 

That was just one more reason we evolved into buying the scheduler feature, just to get the ability to run from command line, and building our own scheduler.  

 

The very nature of MacroReader and AutoHotKey makes them suspect to McAfee and other security products.  If you are a small shop and running without an IT security team that is constantly evolving their software and policies you may be able to use those inexpensive and more basic solutions effectively.  If not then be prepared for the scripts you build to be impacted periodically and your ability to resolve the issues only as effective as the responsiveness of your IT security team...which typically is not very...in most companies they are not designed or encouraged to deal with one-off issues like your script not working because they tightened their policies.