"real time" monitoring of errors in the server environment

[tdiroff]

Hi

The ask is to provide some sort of "real time" status of the Alteryx server environment ... INCLUDING the ability for workflows to successfully communicate with other data sources like NAS. SQL,  Oracle, Teradata, Excel.. etc etc.

We would like to identify potential environment issues before the users start complaining

The thought was to to monitor the number and types of errors being generated in the output of the AS_results collection. We should be able to quickly ID issues that are occurring 

The relevant portion of this collection is only the last X hours where X  may be 1 or 6 or 24 or whatever but definitely not the the entire collection

In a larger corporate environment the As_results collection can be very large ... ours takes 20 min or more to read in but as I mentioned only a small portion of it is relevant to this ask. 

I would like to set a criteria on the mongo db input but nothing is exposed until the service blob is parsed and that means the entire As_results collection is processed

Does anyone have any thoughts on how to accomplish this?

Thank you

Tom D

[mbarone]

I'm not sure if this is helpful in your case, but let me somewhat briefly describe what I did for my situation.

Every morning, I have to check the errors for one specific scheduled workflow that runs overnight.  I need to see if one specific error exists.  Like you, it's not efficient to read in the entire results collection.  I first grab the Queue ID from that specific workflow that ran.  I then created a macro that feeds that Queue ID into the mongo input tool, using the Manual mode:

You may have already known you can do this, and it may or may not help your specific case, but figured I'd throw it out there.

[tdiroff]

HI … I am so sorry for the late response.. I missed your reply...

Since I would be looking in AS_results for a set of workflows that meet a certain criteria ..such "completed in the last hour" .. or something like that ..  I would have to build my As_results query criteria to be dynamic.. based on results from the AS_Queue query.. I will have to think on how to do that...

Tom

[mbarone]

Yep, that's pretty much what I do. 

I have a module that first reads in from the Queue, looks for the module I need, then feeds that Queue ID to the criteria in the screenshot.  Make it a batch macro using the control parameter tool to update that Queue ID in the Mongo results input.

[mbarone · Accepted solution]

The module reads in the Queue, then filters for the module(s) I need.  After I have my Queue IDs the filter tool, I feed them into the macro (red circle):

And that maco just feeds in the Queue IDs to the mongo input tool that scans for the results for those Queue IDs:

[tdiroff]

Perfect!!!

Thank you!!!!

[mbarone]

Welcome!

[raychase]

I use a similar approach to incrementally load Queue/Result data on a 15-minute interval into a SQL environment for further analysis.  Basically I query the max(id) that already exists in my SQL tables, then dynamically uses these values in MongoDB queries to only get records where ids > max(ids).

It's important to note that I only store reporting-relevant data in the SQL environment (to mitigate data volume), and then I can do all kinds of different reporting over this mini-repository without hammering the persistence layer of the application itself.

It's basically a trimmed down version of the AlteryxServerUsageReport with incremental loading capability.

[esoden]

Workflow Admin Manager (WAM), available from Alterxy as an add-on, has deep hooks into Mongo and the APIs and gives you the ability to monitor, report, and alert on almost everything going on inside of Alteryx Server. 

https://capitalizeconsulting.com/workflow-admin-manager/