LAT Log4J Configuration - Ocient Documentation

Data Pipelines are now the preferred method for loading data into the System. For details, see Load Data.

The LAT uses for logging. Users can configure their logging setup however they would like. The LAT packages come with example log4j2.xml files that can be used directly, or modified by the user. This XML code in the log4j2.xml file shows an example configuration.

XML

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" strict="true" monitorInterval="30" packages="biz.paluch.logging.gelf.log4j2">
  <Appenders>
    <RollingRandomAccessFile name="HttpLog" append="true" fileName="logs/http.log" filePattern="logs/http.%d{yyyyMMdd-HH}.%i.log.gz" immediateFlush="true">
      <PatternLayout pattern="%m%n" />
      <Policies>
        <SizeBasedTriggeringPolicy size="100 MB" />
      </Policies>
      <DefaultRolloverStrategy>
        <Delete basePath="logs">
          <IfFileName glob="http.*.log.gz" />
          <IfAccumulatedFileSize exceeds="5 GB" />
        </Delete>
      </DefaultRolloverStrategy>
    </RollingRandomAccessFile>
    <RollingRandomAccessFile name="LATLog" append="true" fileName="logs/lat.log" filePattern="logs/lat.%d{yyyyMMdd-HH}.%i.log.gz" immediateFlush="false">
      <PatternLayout pattern="%d{ISO8601} %-5p [%t] - %m%n" />
      <Policies>
        <SizeBasedTriggeringPolicy size="100 MB" />
      </Policies>
      <DefaultRolloverStrategy>
        <Delete basePath="logs">
          <IfFileName glob="lat.*.log.gz" />
          <IfAccumulatedFileSize exceeds="20 GB" />
        </Delete>
      </DefaultRolloverStrategy>
    </RollingRandomAccessFile>
    <RollingRandomAccessFile name="ErrorLog" append="true" fileName="logs/error.log" filePattern="logs/error.%d{yyyyMMdd-HH}.%i.log.gz" immediateFlush="false">
      <PatternLayout pattern="%m%n" />
      <Policies>
        <SizeBasedTriggeringPolicy size="100 MB" />
      </Policies>
      <DefaultRolloverStrategy fileIndex="nomax">
        <Delete basePath="logs">
          <IfFileName glob="error.*.log.gz" />
          <IfAccumulatedFileSize exceeds="20 GB" />
        </Delete>
      </DefaultRolloverStrategy>
    </RollingRandomAccessFile>
    <!--
    <Gelf name="Graylog" host="udp:graylog-dev" port="12201" version="1.1" extractStackTrace="true">
      <Field name="process_name" literal="lat" />
      <Field name="log_level" pattern="%-5p" />
      <Field name="thread" pattern="%t" />
    </Gelf>
    -->
  </Appenders>
  <Loggers>
    <Logger name="org.apache.kafka" level="info" />
    <Logger name="com.ocient.lat.sink.ocient.Binders.Binder" level="info" />
    <!-- HttpChannel at debug so that jetty preserves the stacktrace when logging uncaught exceptions -->
    <Logger name="org.eclipse.jetty.server.HttpChannel" level="debug" />
    <Logger name="org.eclipse.jetty.server.RequestLog">
      <AppenderRef ref="HttpLog" />
    </Logger>
    <Logger name="com.ocient.lat.sink.file.FileErrorSink" level="info" additivity="false">
      <AppenderRef ref="ErrorLog" />
      <!--
      <AppenderRef ref="Graylog" />
      -->
    </Logger>
    <Logger name="software.amazon.awssdk" level="info" />
    <Logger name="io.netty" level="info" />
    <Root level="info">
      <AppenderRef ref="LATLog" />
      <!--
      <AppenderRef ref="Graylog" />
      -->
    </Root>
  </Loggers>
</Configuration>

Relevant Loggers:

org.apache.kafka: logger.
com.ocient.lat.sink.ocient.Binders.Binder: LAT binder logger. Can be verbose, recommended to keep at info level.
org.eclipse.jetty.server.HttpChannel and org.eclipse.jetty.server.RequestLog: Jetty logs for the HTTP endpoint.
com.ocient.lat.Worker: Main worker logger containing important pipeline logs.

Valid log levels are (in order of least to most verbose):

error
warn
info
debug
trace

Log4j can be configured to log different packages at different levels as shown in the example. Here, the example logs the com.ocient.streaming.client package at the trace level, but the remainder of the code is logged at info level. Ingest Data with Legacy LAT Reference

​Related Links

Related Links