How To: Read Alteryx Server Logs

[khess85]

How To: Read Alteryx Server Logs effectively

Properly reading Alteryx Server logs will significantly help in the resolution of Alteryx Server issues. This article will help you understand and focus your search within the logs.

Prerequisites

• Text editor
  • Recommend: Notepad++ and/or Sublime (This guide will only show Notepad++ but Sublime is similar)
• Alteryx Service and/or Alteryx Gallery logs 

Overview

Alteryx Service Logs (Alteryx Controller)

Alteryx Service Logs are written in the following form (This is expressed at the beginning of every Service log):

Date,LogLevel,ThreadId,LoggerName,ServiceId,RequestId,UserId,RemoteDate,RemoteServer,RemoteProcessId,RemoteThreadId,Message

Each portion of information is separated by a comma.

We will break apart the following job start (If looking for issues with specific jobs or the connection between worker/controller):

2019-11-05 00:52:01.101000,DEBUG,4356,AlteryxService,IP-XXXXXXXX|XXXXXXXX,,AlteryxService,,,,,"SlaveStatusThread: Starting worker  server  queue id "

• Date 2019-11-05 00:52:01.101000
  • The Date the line was written to the log. This is ALWAYS in UTC time. yyyy-mm-dd HH:MM:SS.milliseconds
• LogLevel DEBUG
  • The level at which the log line came from. Can be FATAL, ERROR, DEBUG, INFO, NOTICE, ALERT. These are dependent upon the level of which they have their logging set to in System Settings.
• ThreadId 4356
  • The thread id at the windows level that created the log line, this will not be helpful to us in general troubleshooting
• LoggerName AlteryxService
  • The Name of the logger writing to the log, will be AlteryxService in the Service logs
• ServiceId IP-XXXXXXXX|XXXXXXXX
  • The name of the Service writing to the log. Can be things like RenderQueue, ServiceThreadPool, etc. Giving information about the backend object. In this case, the Hostname of the worker followed by the workerID, seperated with |.
• RequestId
  • Not pertinent in this scope of general diagnosis
• UserId AlteryxService
  • The ID of the user that initially made the request/call in the chain. This can be the ID of a user on gallery, AlteryxService (if internally calling), actual windows username if a schedule through legacy scheduler.
• RemoteDate
  • The date the remote server sent the request (blank if internal calls)
• RemoteServer
  • The IP address of the server(client) making the request (blank if internal calls)
• RemoteProcessId
  • Not pertinent in this scope of general diagnosis but is the processId of the process making the request to the service(blank if internal calls)
• RemoteThreadId
  • Not pertinent in this scope of general diagnosis but is the threadId inside of the process making the request to the service(blank if internal calls)
• Message "SlaveStatusThread: Starting worker server queue id "
  • This is where the bulk of important information will come. Focus on this portion. You will have the output message writing what is being done, trying to be done, the error etc

Alteryx Gallery Logs (Alteryx Gallery)

Alteryx Gallery logs are written in the following form (Expressed at the beginning of every Gallery log):

Date,LogLevel,ThreadId,LoggerName,Location,RequestId,UserId,ClientIP,Server,UserAgent,RequestMethod,RequestTarget,ResponseCode,ResponseTime,Message,Exception

Each portion of information separated by a comma.

We will break apart the following Gallery log line:

2020-01-09 17:55:41.009839,INFO,22,AuthorizationMessageInspector,AfterReceiveRequest,492af65da4de47e8a22290b0301719d1,5cffd92378d23e0be442da5c,<IP ADDRESS>,<MACHINE NAME>,"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/79.0.3945.88 Safari/537.36",GET,/gallery/api/apps/gallery/,,-1,CloudRequest,

• Date 2020-01-09 17:55:41.009839
  • The Date the line was written to the log. This is ALWAYS in UTC time. yyyy-mm-dd HH:MM:SS.milliseconds.
• LogLevel INFO
  • The level at which the log line came from. Can be CRIT, ERROR, FATAL, WARN, DEBUG, INFO. The log level is rarely changed and can only be manually changed by editing the \Alteryx\bin\server\config NLog.config file. The Log Level set in Alteryx System Settings DOES NOT impact the Gallery logs.
• Thread Id 22
  • The thread id at the windows level that created the log line, this will not be helpful to us in general troubleshooting.
• LoggerName AuthorizationMessageInspector
  • The name of the logger creating the log line. This gives a general context to what the back-end was doing.
• Location AfterReceiveRequest
  • Location in the back-end when the line was written to the log. In this case, we received the request and then wrote the line to the log.
• RequestId 492af65da4de47e8a22290b0301719d1
  • The ID of the Request coming in. We will generally not use this in troubleshooting.
• UserId 5cffd92378d23e0be442da5c
  • The userID, or Mongo Document ID, making the request to the gallery. This is useful in tracking a full request or see which user is making the request when multiple users are using the gallery at once.
• ClientIP <IP ADDRESS> EX: 127.0.0.1
  • The IP Address that the request originates from.
• Server <MACHINE NAME> EX: ServerMachineName
  • The hostname of the server that is hosting Alteryx Gallery.
• UserAgent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
  • The "Agent" that a user is using to connect to Alteryx Gallery. This can be several things, including Chrome, Edge, Alteryx Designer (When connecting to Gallery from Designer interface), etc. In this instance you can see this is adhering to a chrome/79 build standard.
• RequestMethod GET
  • The Method of the Request. Can be any HTTP Request Method, like DELETE, POST, GET, etc. GET is retrieving data, POST is trying to create a new piece of data, DELETE is removing a piece of data (asr examples).
• RequestTarget /gallery/api/apps/gallery/
  • The endpoint that the request is trying to access.This is an extremely helpful portion as well as the message portion.This will give you an insight into what we're trying to access and is normally pretty straightforward. Ex. api/scheduler, api/collections, api/apps etc. You will get a solid idea of what we're actually trying to do.
• ResponseCode
  • The HTTP code of the response, ex. 200, 404, 401 etc. In this case it is a request so we will not have a Response code written.
• ResponseTime -1
  • The time it took for the HTTP response, in the case of a request it will be written as -1.
• Message CloudRequest
  • The message that is written giving more context to the log line, what is trying to be done, what is happening (generally), etc. This portion is extremely helpful in diagnosis.
• Exception
  • If there is an error this will hold information about the stack trace of the error and any other information provided by the error. Very helpful in diagnosis.

Procedure

Now that we have an idea of what the logs contain and how each portion could be helpful, on to focusing our attention in them to quickly diagnose the issue!

Initial suggestion:

• Put all pertaining logs into their own folders.

• Doing this allows us to search the entirety of the specific logs sent to us in Notepad++ or Sublime. In Notepad++ you have the "Find in Files" section when using the Search Functionality (CTRL+F).

Now that we have our files in specific sub folders, we can search the entire directory of Service, Gallery, Engine, etc logs for what we are looking for using the "Find in Files" functionality.

1. Enter the Directory path into the "Directory" section. This is where we are searching.
   • NOTE: This will search all sub folders inside the folder path if you have the "In all subfolders" box selected, so deselect if you have additional subfolders you do not want to be searched.
2. Enter what you are searching for in the "Find what:" section.
3. Hit "Find All". You will then be displayed the search results at the bottom of the notepad++ instance like so

• You will notice that what you searched for is inside of the Blue line at the top. This indicates a new search as you can have multiple searches to go back to if necessary. You will also notice the number of "hits" in the number of files (the number of files that were actually searched). This is the number of objects found in the number of files searched.
• You can then double click on the particular line of interest, you will then be displayed the file and shown the exact line inside that file in the main Notepad++ display.

Terms to Search
You may now be asking what exactly to start searching for in which particular logs and that is a fantastic question! My recommendations are as follows to get you started:

• Service logs
  1. Start by searching for FATAL, as these will cause process crashes (not necessarily service crashes but portions of our service will crash) 100% of the time. These are rare but can be easily overlooked.
  2. If no FATAL are found go onto searching for ERROR. These have the possibility of causing process crashes and service crashes.
  3. If no ERROR are found, or not specifically attached to the issue the user is seeing you can then search for WARN.
  4. From there DEBUG, INFO, ALERT and NOTICE are things you can check but normally are informational lines giving you context.
• Gallery logs
  1. Start by searching for CRIT and/or FATAL. These are the same as FATAL in the service logs, it will cause process crashes.
  2. If no CRIT/FATAL then move on to searching for ERROR. These can cause process crashes but will normally give information about what failed and continue running. (Loading circle 404's can be ignored as an example)
  3. If no ERROR then continue onto WARN. These will show information about things that are not necessarily treated as errors but things we can handle and account for in the process.
  4. You can then continue with DEBUG and INFO, these lines are informational and give you surrounding context as to what is happening in the gallery.

The following are some general error lines that you may see (or see something similar) and their meaning. Please note this is not an all inclusive list by any means but meant to give a general starting point to diagnosis.