Log Monitoring

contact {{ocient}} support for assistance before altering the system logging configuration in the rolehostd conf file the {{ocienthyperscaledatawarehouse}} logs events using these severities error warn info verbose debug extended debug by default, the data warehouse creates the text log file /var/opt/ocient/rolehostd log and logs events at the info level and above database logging supports extensive customization and integration with third party tools you can customize using the rolehostd config file the database can return events in text, json, or {{graylog}} extended log format (gelf) formats, or the database can return events in a file, to standard output ( stdout ) , or to a udp port you can customize based on the log source log formats the default log format is text these are some example log entries \[2020 06 09t18 11 25 701594]\[ info]\[role001] rolehost all roles started \[2020 06 09t18 11 25 701881]\[ info]\[role001] rolehost starting all endpoints \[2020 06 09t18 11 25 701925]\[ info]\[role001] rolehost starting endpoint 127 0 0 1 17900 \[2020 06 09t18 11 25 704905]\[ info]\[role001] rolehost starting endpoint 127 0 0 1 17600 \[2020 06 09t18 11 25 708437]\[ info]\[tcpu008] rolehost all endpoints started \[2020 06 09t18 11 25 709236]\[ info]\[role001] host \[2020 06 09t18 11 25 709286]\[ info]\[role001] host rolehostd successfully started \[2020 06 09t18 11 25 709305]\[ info]\[role001] host each log line contains the time, severity, thread, and the source of the log (for example, rolehost or host ) json formatted logs contain the same information but in a structured format {"sys" "oc1 admin0", "timestamp" 1591726285 701594684, "thread" "role001", "lvl" " info", "src" "rolehost", "msg" "all roles started"} {"sys" "oc1 admin0", "timestamp" 1591726285 701881144, "thread" "role001", "lvl" " info", "src" "rolehost", "msg" "starting all endpoints"} {"sys" "oc1 admin0", "timestamp" 1591726285 701925519, "thread" "role001", "lvl" " info", "src" "rolehost", "msg" "starting endpoint 127 0 0 1 17900"} {"sys" "oc1 admin0", "timestamp" 1591726285 704905192, "thread" "role001", "lvl" " info", "src" "rolehost", "msg" "starting endpoint 127 0 0 1 17600"} {"sys" "oc1 admin0", "timestamp" 1591726285 708437413, "thread" "tcpu008", "lvl" " info", "src" "rolehost", "msg" "all endpoints started"} {"sys" "oc1 admin0", "timestamp" 1591726285 709236006, "thread" "role001", "lvl" " info", "src" "host", "msg" " "} {"sys" "oc1 admin0", "timestamp" 1591726285 709286683, "thread" "role001", "lvl" " info", "src" "host", "msg" "rolehostd successfully started"} {"sys" "oc1 admin0", "timestamp" 1591726285 709305246, "thread" "role001", "lvl" " info", "src" "host", "msg" " "} gelf formatted log entries follow the format defined on the gelf via udp page this is an example gelf format {"host" "oc1 admin0", "version" "1 1", "timestamp" 1591726285 701594684, " thread" "role001", "level" "5", " source" "rolehost", "short message" "all roles started"} {"host" "oc1 admin0", "version" "1 1", "timestamp" 1591726285 701881144, " thread" "role001", "level" "5", " source" "rolehost", "short message" "starting all endpoints"} {"host" "oc1 admin0", "version" "1 1", "timestamp" 1591726285 701925519, " thread" "role001", "level" "5", " source" "rolehost", "short message" "starting endpoint 127 0 0 1 17900"} {"host" "oc1 admin0", "version" "1 1", "timestamp" 1591726285 704905192, " thread" "role001", "level" "5", " source" "rolehost", "short message" "starting endpoint 127 0 0 1 17600"} {"host" "oc1 admin0", "version" "1 1", "timestamp" 1591726285 708437413, " thread" "tcpu008", "level" "5", " source" "rolehost", "short message" "all endpoints started"} {"host" "oc1 admin0", "version" "1 1", "timestamp" 1591726285 709236006, " thread" "role001", "level" "5", " source" "host", "short message" " "} {"host" "oc1 admin0", "version" "1 1", "timestamp" 1591726285 709286683, " thread" "role001", "level" "5", " source" "host", "short message" "rolehostd successfully started"} {"host" "oc1 admin0", "version" "1 1", "timestamp" 1591726285 709305246, " thread" "role001", "level" "5", " source" "host", "short message" " "} the database always returns message content in the short message field of the gelf format this table maps ocient severities to gelf system log (syslog) severities ocient severity syslog severity syslog numeric value error error 3 warn warning 4 info notice 5 verbose info 6 debug and extended debug debug 7 log appenders a log appender is the destination for log entries the database can send log entries to a file, a udp port, or to stdout when logs are sent to the file appender, they grow continuously unless you use the truncate option in the configuration you can use tools such as logrotate to manage log files this is a sample logrotate configuration /var/opt/ocient/log/rolehostd log { rotate 5 daily delaycompress compress copytruncate missingok minsize 1m maxsize 100m } test the udp appender using the {{linux}} netcat command $ netcat ul 9000 log configuration an ocient system supports various logging settings to meet different needs set log configuration settings in rolehostd conf , a yaml file normally found in the system /var/opt/ocient/ directory you can change configuration settings to suit your system logging needs in this file for information on checking the status of nodes, see statistics monitoring docid\ jtynlpc rgdksxytttfyh if you are using an ocient simulator, logging behavior and directories might differ from these descriptions for details on how logging works in the ocient simulator, see ocient simulator docid\ pusva0 a2unmu385gcra rolehostd example this is an abbreviated rolehostd conf example in yaml format normally, this file contains other system configurations, but this tutorial focuses on the loggingconfig section, which defines logging behaviors and scopes loggingconfig '' # default scope (all logs) appender logfilepath /path/to/rolehostd log appenders appender allowlevel warn logfilepath /path/to/error log appender allowlevel warn format json logfilepath /path/to/error json dump # dump specific logs allowlevel debug appender logfilepath /path/to/dump log query # query specific logs allowlevel info appender format json logfilepath /path/to/query json security # security specific logs allowlevel info appender format json logfilepath /path/to/security json network logs # example network logging allowlevel info appender type udp server graylog internal port 12201 format gelf compressed true scope definition the top level of the logging configuration is the scope, which determines the corresponding log sources for each configuration setup default scope an empty scope string ( '' ) applies its configuration settings to all log entries the system returns named scopes named scopes apply to log entries where the first component matches the name useful scopes include query — contains information on all queries and statements executed on the system, including users, time, and other details security — contains information on all connections made to the system, successful or unsuccessful dump — contains information on system failures allowlevel the allowlevel parameter specifies which severity levels of logs to capture allowlevel value severity value definitions are debug — most verbose (includes all logs) info — includes informational messages and greater severities verbose — more verbose than the info severity warn — includes warnings and errors only error — includes only error messages appender appenders define the format and directory where the database sends log files you can configure either one or many appenders for each scope single appender for a single appender, use the appender key appender key value \[ ] multiple appenders for multiple output destinations, use the appenders key, followed by a list of appender keys appenders \ appender key value \[ ] \ appender key value \[ ] \[ ] format defines the output format of the log files of an appender appender format format type supported format type values are text — human readable text format (default) json — structured json format gelf — gelf format for integration with graylog logfilepath defines the destination directory for log files of the appender the directory value is a file path for the appender log files, for example /path/to/rolehostd log appender logfilepath directory type determines whether the system submits error logs through stdout or to a separate server using a tcp or udp protocol appender # standard output example type stdout appender # udp example type udp server graylog server example com port 12201 chunksize 1024 # optional compressed true # optional appender # tcp example type tcp server 192 158 1 38 port 12201 server required for tcp or udp types the destination dns name or ip address where the database sends log entries appender server 192 158 1 38 port required for tcp or udp types the port where the database sends log entries the default value is 9000 appender port 12201 chunksize optional for udp type the largest packet size in bytes that the system sends to the server the database discards log entries larger than this value appender chunksize 1024 compressed optional for udp type if you set this option to true , the system compresses packets for details about log compression, see the gelf documentation otherwise, the packets are uncompressed appender compressed true truncate if you set this option to true , the system truncates the log file when the system starts otherwise, the system appends new logs to the file the default value is false appender truncate true related links install an ocient system docid\ ohaih2 yvgdxfvej8iu w set up system monitoring with the tig stack and kapacitor docid\ yrom41vzgnccqriw95tss {{linux}} is the registered trademark of linus torvalds in the u s and other countries