Error Monitoring

The

By default the data warehouse will create a text log file in /var/opt/ocient/rolehostd.log and log events at Info level and above.

Database logging supports extensive customization and integration with third party tools. Customization is done through the rolehostd config file. Events can be output in text, json, and

The default log format is text, and example log entries are as follows:

Each log line contains the time, severity, thread, and the source of the log (for example "roleHost" or "host" in this example).

JSON formatted logs contain the same information, but in a structured format:

And finally, GELF formatted log entries follow the format defined on the GELF via UDP page and look as follows:

The message content is currently always delivered in the "short_message" field of the GELF format.



A log appender is the destination for log entries. Log entries can be sent to a file, a UDP port, or to stdout.

When logs are sent to the file appender, they will grow continuously, unless the "truncate" option is used in the configuration. Tools such as logrotate can be used to manage the log files.

This is a sample logrotate configuration.

Test the UDP appender using a Linux netcat command similar to this one.

Logs are configured in the rolehostd configuration file (/var/opt/ocient/rolehostd.conf by default) in YAML format. A sample rolehostd.conf file is:

Under the "loggingConfig" section is a list of log sources (e.g., "etlJobService_t") and their configuration. The "root" source is the empty string ("" in this example). In this example, the root source has an "allowLevel" of info, which says to log errors at info level and above. It has a primary appender of type file, that will not be truncated every time the database starts, and whose path is "/var/opt/ocient/logs/rolehostd.log".

The root source also has a secondary appender, under the "appenders" section of the YAML. In addition to writing log entries to the log file, it will also send log entries in gelf format to the server graylog.example.com at port 12201 over UDP. The "allowLevel" of the secondary appender is "error", so only entries of error and above will be sent to the GELF server.

In addition to a configuration for the root source, there is also a configuration for the "etlJobService_t" source. That will log all entries of debug level and above that source to the file /var/opt/ocient/logs/etl.log. By default, log sources are hierarchical. In other words, entries from the source "a::b::c" will also be sent to "a::b" and "a" and "", or in this example, entries from "etlJobService_t" would also written to the /var/opt/ocient/logs/rolehostd.log file configured for the root source. However the etlJobService_t source is configured with "exclusive: True", so entries will only be written to the etl.log file, and not to things higher up in the hierarchy. This example also sends entries for queries and security to their own log files, in JSON format.

The path to the root appender log file can be overridden by the --log-file argument to the rolehostd program.

The codebase contains a VERBOSE-level logger called protocol::systemMetadata which logs the entire system metadata object every time it is updated. Because the default log level is INFO, the object will not be included in the main logs. To start logging the system metadata object to a file /some/log/file.txt, the user can add the following to rolehostd.conf:

The line exclusive: true is included so that the system object does not clutter the main log files as well. The user will need to restart rolehostd for their changes to take effect.

Log configuration is in the top level loggingConfig section of the configuration YAML file. Under loggingConfig is a list of log sources, which are strings matching the first part of each log line (e.g., etlJobService_t). The empty string log source is the root log source that matches all log messages. Log sources are hierarchical, so a log source of "a::b" will additionally get sent to the configuration for "a" and for "" (if they exist).

Each log source can have the following entries:

allowLevel The log level assigned for this source. This can be either error, info, warn, verbose, debug, or edebug. Defaults to info.

exclusive True or False indicating whether log entries should be sent to sources higher up the hierarchy. Defaults to False.

appender The primary appender source for this log source.

appenders A list of one or more secondary appenders.

Each appender can have the following configuration values:

type One of file, stdout, udp, or tcp.

format One of text, json, or gelf.

allowLevel Secondary appenders can have an additional allowLevel. This further filters the allowLevel specified under the log source. In other words, the log source might have an allowLevel of info, but a secondary appender might further filter entries to only error. The allowLevel in the secondary appender should be greater than the allowLevel for the log source.

hostname For json and gelf format entires, the "sys" or "host" values respectively. If not specified, the actual hostname of the system will be used. This allows for a more descriptive hostname to be specified if required.

If the type of the appender is "file", the following configuration entries can exist:

logFilePath The path to the log file.

truncate True or False, whether the log file should be truncated when the program starts.

If the type of the appender is "udp" or "tcp", the following configuration entries can exist:

server The DNS name or IP address of the server where the log entries will be sent.

port The port where the log entries will be sent. Defaults to 9000.

chunkSize (UDP only) The largest packet that will be sent. Log entries larger than this will be discarded. Note that GELF supports splitting log entries across UDP packets, but that is not currently implemented.

Install an Ocient System

Set Up System Monitoring with the TIG Stack and Kapacitor