Improve Log files for simpler parsing

[AndrewW]

To get simple information from a workflow, such as the name, run start date/time and run end date/time is far more complex than it should be. Ideally the log, in separate line items distinctly labelled, would have the workflow path & name, the start date/time, and end date/time and potentially the run time to save having to do a calculation. Also having an overall module status would be of use, i.e. if there was an Error in the run the overall status is Error, if there was a warning the overall status is Warning otherwise Success.

Parsing out the workflow name and start date/time is challenge enough, but then trying to parse out the run time, convert that to a time and add it to the start date/time to get the end date/time makes retrieving basic monitoring information far more complex than it should be.

[Hugo]

Having the file name in the title would be great too

[fharper]

I think it useful to point out that log files can exist on two levels, at least I am only aware of 2.

The first is system generated log files with a sequential number embedded to maintain uniqueness like "Engine x64_Log_1304355770.log"  and puts these in a folder you can configure or is defaulted.  

The second is the output view you see when running manually or that you capture when running from a batch file and redirecting output to a log file you define.

The first is certainly difficult to begin with because you can't easily identify by file name what flow or flow instance it refers to.  so a naming convention based on flow name and timestamp would be very helpful.  after that parsing and enhancements to content are basically the same.

I wrote a parser for log files and found it not difficult and there is a parser macro in the Crew macros, found after I wrote my own..., but I do see value in standardizing the output in some areas aside from system generated log files having meaningful names.

• key info should have standard prefixes or Tag strings we can parse on as well as easily spot visually, flow path, flow name, start and end times, duration, maybe a few others.
  • For example I see the flow being run in full path with a standard prefix "Started running" in position 1 of row but I parse out the flow name from the path for analysis and reporting/notification purposes.  If the log posted some of the Engine constants like [Engine.TempFilePath], [Engine.WorkflowDirectory], [Engine.WorkflowFileName] then we would have info pre-parsed to a degree and ideally all in one area of messaging.
• I know with the message tool, which I have started using, allows you to choose to a degree where/when it will appear in the stream of the output view, which we capture when running jobs from batch files to create logs.  From this I perceive the log is actually a composite of possibly several temp log streams.  If possible it would be nice to have the basic stats up front or at end, the point being in one place for a nice visual snapshot without scrolling up and down to pick these things out.  Not so much for parsing but for human eye.  Flow name, path, start, end, duration, rows read or written by tool all in one place and no jumping around.  Then let the current flow of messages happen as is so we can analyze order of events if needed.  This may be easy to do as a separate tracking log summary which is appended or prepended to the existing log presentation. 
• Tangent thought...in the Message tool we can set the type of message and level to determine how it is formatted and if it resolves to the final output view if in a macro..If Alteryx does create a "Run Summary" section this would in effect create a new "type" and thus adds value to adding to the message tool to cause those messages flag as such to be formatted and grouped to that section of output view messaging.
•  All of my thoughts above are still largely targeted toward a program parsing the data but there is another idea that may align well with Andrew's original request and that is for Alteryx to take the summary type info I spoke of plus any other elements others may add and create a single row "summary" log file with a name composed of flow name and timestamp that is a yxdb and has columns for the various key fields like flow path, flow name, start and end times and duration and high level error level.  This allows someone who does not need a deeper analysis to read a small file and see basic run info without parsing and see if it failed or not and drive reporting/alerts.
  • If they need more then they have to write a parser for more complex analysis and reporting.

The way we operationalize/automate our "production work" we need to know more than summary info, we analyze errors and warnings and analyze those messages for known and unknown reasons to determine if a failed job is actually ok or not and if it should be auto-restarted or a person responsible notified for research, etc.  for example if the job fails for known messages we associate with network or "system" issues, like DB connection lost, our scheduler will auto-restart every 10 minutes up to a specified number of attempts before alerting the on call person.  another case is when a flow using R-code generates a certain error, we know it is a "false" error from known issue with R-Code and we change the flag in our scheduler from failed to success and don't alert or restart.

So for us tweaks to the content and organization as mentioned are most appealing and beneficial to all but the summary log file idea may be a big win for Andrew and others. 

[derekbelyea]

Some further ideas for enhanced log functionality:

• Provide user setting options for lite, standard, verbose and no logging
• Make consistent use of delimiters within log files or better yet, save log files using JSON formatting
• Tag event and message types for easier parsing
• For verbose logging have options to:
  • Include Tool Name as well as ToolID
  • Include user settings
  • Include workflow configuration settings and values
  • Include tool settings