Alerting on Workflow Error - best practices?

[Scott_Snowman]

Hello all, I am trying to improve visibility and action on errors in scheduled workflows in the Gallery.

Currently we do one of two things when a scheduled workflow errors:

1. Have the workflow send an email after "Run with Errors" to an escalation mailbox
2. We regularly collect the results of the run by sifting through the Gallery's MongoDB log and display the results in a visualization platform

#1 is meant to drive calls to action while #2 is meant to help understand trends of failures over time.

We'd like to get #1 even more real time and out of email by connecting to a Slack webhook or a Jira POST request or the like. However it doesn't appear you can easily pass information out of Alteryx via the Workflow Event architecture.

Are there other best practices users have found success with to help escalate workflow errors?

[cmcclellan]

If you want to integrate with Slack or Jira, why not write a macro that calls a Python routine to do that ?  That would be instant notification when the macro is executed.

[Scott_Snowman]

Hi @cmcclellan I'd considered that, but I'd love to be able to pass the error log into Jira or Slack. Unfortunately, the only way I have found to do so is using the Email After Workflow Run event architecture.

I'd settle for not having access to the entire log, but I'd at least want to have basic metadata such as the workflow's name available to the Python process.  I haven't found any way to get that specific detail out of Alteryx either, except for explicitly hard-coding it into the process that is kicked off post-run. (Which of course necessitates creating a separate copy of the process for each scheduled workflow.)

Any ideas on getting that detail to the Python process?

[cmcclellan]

You can get the workflow name from [Engine.WorkflowFileName] and the error log might be possible (I used to just get the latest file in the log directory because we were only running 1 workflow at a time and it was simple to work out).  I'm sure you could pull the error log from somewhere 🙂 

[danilang]

Hi @Scott_Snowman 

Any suggestion built using only Alteryx is going to fail if the workflow errors before the log reporting part has run.  There won't be any results.  One thing you can look into is using the CReW Runner macro.   This macro runs the workflow out-of-process so the main workflow can fail without affecting the log reporting part.  The macro has two outputs, one that returns the results if the workflow is successful and the other if the workflow fails.   After the workflow called by the macro has run, you can log the results to Jira or Slack using their respective APIs.  If the called workflow has failed, you can issue an alert by either email or any other way that an Alteryx workflow can trigger.

Dan

[Scott_Snowman]

Thanks @danilang -- to be more specific, rather than simply the workflow erroring out (and not finishing its run) I'm looking at a way to tap into the detail in the Error Log available to the native "After Run with Errors" event.

I'll look into the CReW macro suite. Based on the description at http://www.chaosreignswithin.com/2014/09/blog-macro-pack-2014-q3-release.html it looks like it will do exactly what we are looking for in that it outputs the results of the error log.

Thank you!