LAT Metrics

data pipelines are now the preferred method for loading data into the ocient system for details, see load data docid 1nw163anqyhk58cqmepjs configuration lat metrics are exposed using the lat rest api endpoints lat exposes two metrics endpoints; a partitions endpoint, and a pipeline endpoint partition metrics the partitions endpoint exposes individual metrics for every partition that is participating in the current lat instance for {{kafka}} loads, partitions correspond to kafka’s partitions for file source based loads, partitions correspond to the independent file sets created within each file group for parallel processing the partitions metrics endpoint can be accessed by issuing a curl request as follows curl http //127 0 0 1 8080/v2/metrics/read/lat\ type=partitions a response for this request will look as follows, with one entry per partition in the partitions array { "request" { "mbean" "lat\ type=partitions", "type" "read" }, "value" { "partitions" \[ { "offsets durable" 0, "pushes errors" 0, "pushes attempts" 0, "rows pushed" 0, "offsets written" 0, "records buffered" 0, "records errors column" 0, "records errors deserialization" 0, "records errors transformation" 0, "offsets processed" 0, "lag" 0, "partition" "topic 0", "records filter accepted" 0, "records errors row" 0, "records filter rejected" 0, "records errors generic" 0, "producer send attempts" 0, "offsets pushed" 0, "pushes unacknowledged" 0, "invalid state" 0, "bytes pushed" 0, "errors partition" 0, "records errors total" 0, "offsets buffered" 0, "complete" 0, "offsets end" 0, "producer send errors" 0 } ] }, "timestamp" 1642497992, "status" 200 } partition metrics definitions metrics definitions are as follows metric description partition topic partition pair (e g , mytopic 0 ) records errors deserialization total number of processed records which failed to deserialize records errors transformation total number of processed records which failed to transform records errors column total number of processed records which failed to bind transformed values to columns records errors row total number of processed records which failed during builder row advancement/other builder errors records errors generic total number of processed records which failed for any other reason records errors total sum of all record errors for this topic records buffered total number of processed records which have not yet been pushed should always be modulo configured buffer size records filter rejected number of records rejected by the topic filter records filter accepted number of records accepted by the topic filter offsets processed for kafka loading, records are processed if their offsets are strictly less than the value of offsets processed for file loading, this represents the index of the most recently processed file if there is an error during the processing of a record/file, this metric is still updated according to that record/file’s offset when a file load is complete, this offset becomes equal to offsets end can decrease due to reprocessing offsets written for kafka loading, records are written if their offsets are strictly less than the value of offsets written for file loading, this represents the index of the most recently written file this metric is only updated when the record/file is processed and the write is successful note that a successful write does not mean the record/file is durable can decrease due to reprocessing offsets buffered for kafka loading, this represents the most recently processed record which was processed to completion and is waiting to be pushed into the {{ocienthyperscaledatawarehouse}} for file loading, this represents the index of the most recently processed file which was processed to completion and is waiting to be pushed into the {{ocient}} data warehouse offsets pushed for kafka loading, this represents the highest offset of the batch of rows most recently pushed to the ocient data warehouse for file loading, this represents the highest file index of the batch of rows most recently pushed to the ocient data warehouse can decrease due to reprocessing offsets end for kafka loading, this represents the end offset of the partition for file loading, this represents the file count of the partition should never decrease offsets durable for kafka loading, records are made durable if their offsets are strictly less than the value of offsets durable for file loading, this represents the most durable file index should never decrease producer send attempts number of kafka error topic producer send attempts producer send errors number of kafka error topic producer send errors bytes pushed number of bytes pushed into the ocient data warehouse rows pushed number of rows pushed into the ocient data warehouse pushes attempts number of attempts to push record batches into the ocient data warehouse for this partition pushes errors number of attempts to push record batches into the ocient data warehouse which resulted in error pushes unacknowledged number of attempts to push record batches into the ocient data warehouse for which no response has yet been received invalid state number of times a code path was reached in lat which is erroneous complete whether or not the partition has any records left to process at the moment; this status can change often in a kafka load but will likely not change from complete to incomplete in a file load errors partition the number of times the lat failed to fetch records for a particular partition lag calculated as offsets end offsets durable; in a kafka load, lag represents the number of unprocessed records in a file load, lag represents the number of unprocessed files lag becomes zero when the complete status is true pipeline metrics the pipeline endpoint exposes aggregate metrics for the lat instance there are three categories of pipeline metrics pipeline metric category description "pipeline" aggregate monotonically increasing metrics that act as lifetime counters for the running pipeline they are independent of the current set of partitions for instance, rows pushed is the total number of rows ever pushed for this pipeline "partitions" aggregate metrics that depend on the current set of active partitions for the running pipeline for instance, total lag is the summed lag across all currently active partitions for the pipeline pipeline specific pipeline only metrics that are not aggregates the pipeline metrics endpoint can be accessed by issuing a curl request as follows curl http //127 0 0 1 8080/v2/metrics/read/lat\ type=pipeline a response for this request will look as follows { "request" { "mbean" "lat\ type=pipeline", "type" "read" }, "value" { "pipeline" { "pushes errors" 0, "pushes attempts" 0, "rows pushed" 0, "records errors column" 0, "records errors deserialization" 0, "records errors transformation" 0, "bytes pushed" 0, "errors partition" 0, "records filter accepted" 0, "records errors total" 0, "records errors row" 0, "records filter rejected" 0, "records errors generic" 0, "producer send attempts" 0, "producer send errors" 0 }, "partitions" { "max lag" 0, "total offsets buffered" 0, "avg lag" 0 0, "total pushes unacknowledged" 0, "min lag" 0, "total lag" 0, "total invalid state" 0, "total offsets pushed" 0, "total records buffered" 0, "total complete" 0, "total offsets processed" 0 }, "paused" 0, "bytes buffered" 0, "complete" 0, "workers" 32, "lat version" "3 0 0" }, "timestamp" 1642497956, "status" 200 } individual metrics such as lag , offsets buffered , pushes errors , etc are defined within the lat metrics docid\ rxnrtpucwkuw8ter7hvsl section however, metrics specific only to the pipeline endpoint are defined as follows metric description paused 0 if processing is active across all workers, 1 otherwise bytes buffered global allocated memory in bytes complete 1 if the pipeline has finished the entire loading process, 0 otherwise workers the number of active workers lat version the version of the running lat server related links ingest data with legacy lat reference docid ul5nilgyd7y wjccg59d system information rest endpoints docid\ r kngib6smibxnwafdiq