> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ocient.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Log Monitoring

export const OcientDataIntelligencePlatform = "OcientAIQ™ Unified Data Platform";

export const Ocient = "Ocient®";

export const Linux = "Linux®";

export const Graylog = "Graylog®";

<Warning>
  Contact {Ocient} Support for assistance before altering the system logging configuration in the `rolehostd.conf` file.
</Warning>

The {OcientDataIntelligencePlatform} logs events using these severities:

* `Error`
* `Warn`
* `Info`
* `Verbose`
* `Debug`
* `Extended Debug`

By default, the data warehouse creates the text log file `/var/opt/ocient/rolehostd.log` and logs events at the `Info` level and above.

Database logging supports extensive customization and integration with third-party tools. You can customize using the `rolehostd` config file. The database can return events in text, JSON, or {Graylog} Extended Log Format (GELF) formats, or the database can return events in a file, to standard output (`stdout`), or to a UDP port. You can customize based on the log source.

## Log Formats

The default log format is text. These are some example log entries.

```Text Text theme={null}
[2020-06-09T18:11:25.701594][ INFO][role001] roleHost: All roles started
[2020-06-09T18:11:25.701881][ INFO][role001] roleHost: Starting all endpoints
[2020-06-09T18:11:25.701925][ INFO][role001] roleHost: Starting endpoint 127.0.0.1:17900
[2020-06-09T18:11:25.704905][ INFO][role001] roleHost: Starting endpoint 127.0.0.1:17600
[2020-06-09T18:11:25.708437][ INFO][tcpU008] roleHost: All endpoints started
[2020-06-09T18:11:25.709236][ INFO][role001] host: ------------------------------
[2020-06-09T18:11:25.709286][ INFO][role001] host: Rolehostd successfully started
[2020-06-09T18:11:25.709305][ INFO][role001] host: ------------------------------
```

Each log line contains the time, severity, thread, and the source of the log (for example, `roleHost` or `host`).

JSON-formatted logs contain the same information but in a structured format.

```json JSON theme={null}
{"sys":"oc1.admin0", "timestamp":1591726285.701594684, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"All roles started"}
{"sys":"oc1.admin0", "timestamp":1591726285.701881144, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"Starting all endpoints"}
{"sys":"oc1.admin0", "timestamp":1591726285.701925519, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"Starting endpoint 127.0.0.1:17900"}
{"sys":"oc1.admin0", "timestamp":1591726285.704905192, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"Starting endpoint 127.0.0.1:17600"}
{"sys":"oc1.admin0", "timestamp":1591726285.708437413, "thread":"tcpU008", "lvl":" INFO", "src":"roleHost", "msg":"All endpoints started"}
{"sys":"oc1.admin0", "timestamp":1591726285.709236006, "thread":"role001", "lvl":" INFO", "src":"host", "msg":"------------------------------"}
{"sys":"oc1.admin0", "timestamp":1591726285.709286683, "thread":"role001", "lvl":" INFO", "src":"host", "msg":"Rolehostd successfully started"}
{"sys":"oc1.admin0", "timestamp":1591726285.709305246, "thread":"role001", "lvl":" INFO", "src":"host", "msg":"------------------------------"}
```

GELF-formatted log entries follow the format defined on the [GELF via UDP](https://go2docs.graylog.org/5-0/getting_in_log_data/gelf.html) page. This is an example GELF format.

```json JSON theme={null}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.701594684, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"All roles started"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.701881144, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"Starting all endpoints"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.701925519, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"Starting endpoint 127.0.0.1:17900"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.704905192, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"Starting endpoint 127.0.0.1:17600"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.708437413, "_thread":"tcpU008", "level":"5", "_source":"roleHost", "short_message":"All endpoints started"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.709236006, "_thread":"role001", "level":"5", "_source":"host", "short_message":"------------------------------"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.709286683, "_thread":"role001", "level":"5", "_source":"host", "short_message":"Rolehostd successfully started"}
{"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.709305246, "_thread":"role001", "level":"5", "_source":"host", "short_message":"------------------------------"}
```

The database always returns message content in the `short_message` field of the GELF format.

This table maps Ocient severities to GELF system log (syslog) severities.

| **Ocient Severity**          | **Syslog Severity** | **Syslog Numeric Value** |
| ---------------------------- | ------------------- | ------------------------ |
| `Error`                      | `Error`             | 3                        |
| `Warn`                       | `Warning`           | 4                        |
| `Info`                       | `Notice`            | 5                        |
| `Verbose`                    | `Info`              | 6                        |
| `Debug` and `Extended Debug` | `Debug`             | 7                        |

## Log Appenders

A log appender is the destination for log entries. The database can send log entries to a file, a UDP port, or to `stdout`.

When logs are sent to the file appender, they grow continuously unless you use the `truncate` option in the configuration. You can use tools such as `logrotate` to manage log files.

This is a sample `logrotate` configuration.

```Text Text theme={null}
/var/opt/ocient/log/rolehostd.log {
   rotate 5
   daily
   delaycompress
   compress
   copytruncate
   missingok
   minsize 1M
   maxsize 100M
}
```

Test the UDP appender using the {Linux} `netcat` command.

```linux LINUX theme={null}
$ netcat -ul 9000
```

## Log Configuration

An Ocient System supports various logging settings to meet different needs.

Set log configuration settings in `rolehostd.conf`, a YAML file normally found in the system `/var/opt/ocient/` directory. You can change configuration settings to suit your system logging needs in this file.

For information on checking the status of nodes, see [Statistics Monitoring](/statistics-monitoring).

<Info>
  If you are using an Ocient Simulator, logging behavior and directories might differ from these descriptions.

  For details on how logging works in the Ocient Simulator, see [Simulator Logging](/ocient-simulator#simulator-logging).
</Info>

`rolehostd` **Example**

This is an abbreviated `rolehostd.conf` example in YAML format. Normally, this file contains other system configurations, but this tutorial focuses on the `loggingConfig` section, which defines logging behaviors and scopes.

```yaml YAML theme={null}
loggingConfig:
    '':  # Default scope (all logs)
        appender:
        logFilePath: /path/to/rolehostd.log
    appenders:
    - appender:
        allowLevel: warn
        logFilePath: /path/to/error.log
    - appender:
        allowLevel: warn
        format: json
        logFilePath: /path/to/error.json
  
    dump:  # Dump-specific logs
        allowLevel: debug
        appender:
            logFilePath: /path/to/dump.log
  
    query:  # Query-specific logs
        allowLevel: info
        appender:
            format: json
            logFilePath: /path/to/query.json
  
    security:  # Security-specific logs
        allowLevel: info
        appender:
            format: json
            logFilePath: /path/to/security.json

    network-logs:  # Example network logging
        allowLevel: info
        appender:
            type: UDP
            server: graylog.internal
            port: 12201
            format: gelf
            compressed: true
```

### **Scope Definition**

The top level of the logging configuration is the scope, which determines the corresponding log sources for each configuration setup.

**Default Scope**

An empty scope string (`''`) applies its configuration settings to all log entries the system returns.

**Named Scopes**

Named scopes apply to log entries where the first component matches the name. Useful scopes include:

* `query` — Contains information on all queries and statements executed on the system, including users, time, and other details.
* `security` — Contains information on all connections made to the system, successful or unsuccessful.
* `dump` — Contains information on system failures.

### `allowLevel`

The `allowLevel` parameter specifies which severity levels of logs to capture.

```yaml YAML theme={null}
allowLevel: value
```

Severity `value` definitions are:

* `debug` — Most verbose (includes all logs). 
* `info` — Includes informational messages and greater severities. 
* `verbose` — More verbose than the `info` severity.  
* `warn` — Includes warnings and errors only. 
* `error` — Includes only error messages. 

### `appender`

Appenders define the format and directory where the database sends log files. You can configure either one or many appenders for each scope.

**Single Appender**

For a single appender, use the `appender` key.

```yaml YAML theme={null}
appender:
    key: value
    [ ... ]
```

**Multiple Appenders**

For multiple output destinations, use the `appenders` key, followed by a list of `appender` keys.

```yaml YAML theme={null}
 appenders:
 - appender:
     key: value
     [ ... ]
 - appender:
     key: value
     [ ... ]
 [ -...]
```

### `format`

Defines the output format of the log files of an appender.

```yaml YAML theme={null}
appender:
    format: format_type
```

Supported `format_type` values are:

* `text` — Human-readable text format (default). 
* `json` — Structured JSON format.
* `gelf` — GELF format for integration with Graylog.

### `logFilePath`

Defines the destination directory for log files of the appender. The `directory` value is a file path for the appender log files, for example: `/path/to/rolehostd.log`.

```yaml YAML theme={null}
appender:
    logFilePath: directory
```

### `type`

Determines whether the system submits error logs through `stdout` or to a separate server using a TCP or UDP protocol.

```yaml YAML theme={null}
appender: # standard output example
    type: stdout

appender: # UDP example
    type: UDP
    server: graylog-server.example.com
    port: 12201
    chunkSize: 1024  # Optional
    compressed: true  # Optional

appender: # TCP example
    type: TCP
    server: 192.158.1.38
    port: 12201
```

`server`

<Info>
  Required for TCP or UDP types.
</Info>

The destination DNS name or IP address where the database sends log entries.

```yaml YAML theme={null}
appender:
    server: 192.158.1.38
```

`port`

<Info>
  Required for TCP or UDP types.
</Info>

The port where the database sends log entries. The default value is `9000`.

```yaml YAML theme={null}
appender:
    port: 12201
```

`chunkSize`

<Info>
  Optional for UDP type.
</Info>

The largest packet size in bytes that the system sends to the server. The database discards log entries larger than this value.

```yaml YAML theme={null}
appender:
    chunkSize: 1024
```

`compressed`

<Info>
  Optional for UDP type.
</Info>

If you set this option to `true`, the system compresses packets. For details about log compression, see the [GELF documentation](https://go2docs.graylog.org/current/getting_in_log_data/gelf.html). Otherwise, the packets are uncompressed.

```yaml YAML theme={null}
appender:
    compressed: true
```

### `truncate`

If you set this option to `true`, the system truncates the log file when the system starts. Otherwise, the system appends new logs to the file. The default value is `false`.

```yaml YAML theme={null}
appender:
    truncate: true
```

### `redactions`

You can redact data in logs at the SQL statement level using the optional `redactions` configuration. Enabling redaction creates another version of the logs with redacted SQL text. The `redactions` configuration value is `statement` for the redaction of part of the statement.

```yaml YAML theme={null}
appender:
    redactions:
    - statement # Optional
```

`rolehostd` **Log Redaction Example**

This abbreviated `rolehostd.conf` file configuration enables log redaction at the statement level for both the `rolehostd.log` and `query.json` log files.

```yaml YAML theme={null}
loggingConfig:
    ? '':
    appenders:
    - appender:
        logFilePath: /var/opt/ocient/logs/rolehostd.log
    - appender:
        logFilePath: /var/opt/ocient/logs/rolehostd_redacted.log
        redactions:
        - statement
    query:
        allowLevel: info
        appenders:
        - appender:
            format: json
            logFilePath: /var/opt/ocient/logs/query.json
        - appender:
            format: json
            logFilePath: /var/opt/ocient/logs/query_redacted.json
            redactions:
            - statement
```

## Related Links

[Install an Ocient System](/install-an-ocient-system)

[Set Up System Monitoring with the TIG Stack and Kapacitor](/set-up-system-monitoring-with-the-tig-stack-and-kapacitor)

***

*Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.*