Documentation Index
Fetch the complete documentation index at: https://docs.ocient.com/llms.txt
Use this file to discover all available pages before exploring further.
Data Pipelines are now the preferred method for loading data into the System. For details, see Load Data. log4j2.xml files that can be used directly, or modified by the user. This XML code in the log4j2.xml file shows an example configuration.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" strict="true" monitorInterval="30" packages="biz.paluch.logging.gelf.log4j2">
<Appenders>
<RollingRandomAccessFile name="HttpLog" append="true" fileName="logs/http.log" filePattern="logs/http.%d{yyyyMMdd-HH}.%i.log.gz" immediateFlush="true">
<PatternLayout pattern="%m%n" />
<Policies>
<SizeBasedTriggeringPolicy size="100 MB" />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="logs">
<IfFileName glob="http.*.log.gz" />
<IfAccumulatedFileSize exceeds="5 GB" />
</Delete>
</DefaultRolloverStrategy>
</RollingRandomAccessFile>
<RollingRandomAccessFile name="LATLog" append="true" fileName="logs/lat.log" filePattern="logs/lat.%d{yyyyMMdd-HH}.%i.log.gz" immediateFlush="false">
<PatternLayout pattern="%d{ISO8601} %-5p [%t] - %m%n" />
<Policies>
<SizeBasedTriggeringPolicy size="100 MB" />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="logs">
<IfFileName glob="lat.*.log.gz" />
<IfAccumulatedFileSize exceeds="20 GB" />
</Delete>
</DefaultRolloverStrategy>
</RollingRandomAccessFile>
<RollingRandomAccessFile name="ErrorLog" append="true" fileName="logs/error.log" filePattern="logs/error.%d{yyyyMMdd-HH}.%i.log.gz" immediateFlush="false">
<PatternLayout pattern="%m%n" />
<Policies>
<SizeBasedTriggeringPolicy size="100 MB" />
</Policies>
<DefaultRolloverStrategy fileIndex="nomax">
<Delete basePath="logs">
<IfFileName glob="error.*.log.gz" />
<IfAccumulatedFileSize exceeds="20 GB" />
</Delete>
</DefaultRolloverStrategy>
</RollingRandomAccessFile>
<!--
<Gelf name="Graylog" host="udp:graylog-dev" port="12201" version="1.1" extractStackTrace="true">
<Field name="process_name" literal="lat" />
<Field name="log_level" pattern="%-5p" />
<Field name="thread" pattern="%t" />
</Gelf>
-->
</Appenders>
<Loggers>
<Logger name="org.apache.kafka" level="info" />
<Logger name="com.ocient.lat.sink.ocient.Binders.Binder" level="info" />
<!-- HttpChannel at debug so that jetty preserves the stacktrace when logging uncaught exceptions -->
<Logger name="org.eclipse.jetty.server.HttpChannel" level="debug" />
<Logger name="org.eclipse.jetty.server.RequestLog">
<AppenderRef ref="HttpLog" />
</Logger>
<Logger name="com.ocient.lat.sink.file.FileErrorSink" level="info" additivity="false">
<AppenderRef ref="ErrorLog" />
<!--
<AppenderRef ref="Graylog" />
-->
</Logger>
<Logger name="software.amazon.awssdk" level="info" />
<Logger name="io.netty" level="info" />
<Root level="info">
<AppenderRef ref="LATLog" />
<!--
<AppenderRef ref="Graylog" />
-->
</Root>
</Loggers>
</Configuration>
org.apache.kafka: logger.com.ocient.lat.sink.ocient.Binders.Binder: LAT binder logger. Can be verbose, recommended to keep at info level.org.eclipse.jetty.server.HttpChannel and org.eclipse.jetty.server.RequestLog: Jetty logs for the HTTP endpoint.com.ocient.lat.Worker: Main worker logger containing important pipeline logs.
Valid log levels are (in order of least to most verbose):
errorwarninfodebugtrace
Log4j can be configured to log different packages at different levels as shown in the example. Here, the example logs the com.ocient.streaming.client package at the trace level, but the remainder of the code is logged at info level.
Ingest Data with Legacy LAT Reference
