LAT Metrics

data pipelines are now the preferred method for loading data into the ocient system for details, see docid\ zncvnrhsf6fg1yvqk6mxt configuration lat metrics are exposed using the lat rest api endpoints lat exposes two metrics endpoints; a partitions endpoint, and a pipeline endpoint partition metrics the partitions endpoint exposes individual metrics for every partition that is participating in the current lat instance for {{kafka}} loads, partitions correspond to kafka’s partitions for file source based loads, partitions correspond to the independent file sets created within each file group for parallel processing the partitions metrics endpoint can be accessed by issuing a curl request as follows curl http //127 0 0 1 8080/v2/metrics/read/lat\ type=partitions a response for this request will look as follows, with one entry per partition in the partitions array { "request" { "mbean" "lat\ type=partitions", "type" "read" }, "value" { "partitions" \[ { "offsets durable" 0, "pushes errors" 0, "pushes attempts" 0, "rows pushed" 0, "offsets written" 0, "records buffered" 0, "records errors column" 0, "records errors deserialization" 0, "records errors transformation" 0, "offsets processed" 0, "lag" 0, "partition" "topic 0", "records filter accepted" 0, "records errors row" 0, "records filter rejected" 0, "records errors generic" 0, "producer send attempts" 0, "offsets pushed" 0, "pushes unacknowledged" 0, "invalid state" 0, "bytes pushed" 0, "errors partition" 0, "records errors total" 0, "offsets buffered" 0, "complete" 0, "offsets end" 0, "producer send errors" 0 } ] }, "timestamp" 1642497992, "status" 200 } partition metrics definitions metrics definitions are as follows 284,332 true unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type unhandled content type pipeline metrics the pipeline endpoint exposes aggregate metrics for the lat instance there are three categories of pipeline metrics 154,388 true left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type the pipeline metrics endpoint can be accessed by issuing a curl request as follows curl http //127 0 0 1 8080/v2/metrics/read/lat\ type=pipeline a response for this request will look as follows { "request" { "mbean" "lat\ type=pipeline", "type" "read" }, "value" { "pipeline" { "pushes errors" 0, "pushes attempts" 0, "rows pushed" 0, "records errors column" 0, "records errors deserialization" 0, "records errors transformation" 0, "bytes pushed" 0, "errors partition" 0, "records filter accepted" 0, "records errors total" 0, "records errors row" 0, "records filter rejected" 0, "records errors generic" 0, "producer send attempts" 0, "producer send errors" 0 }, "partitions" { "max lag" 0, "total offsets buffered" 0, "avg lag" 0 0, "total pushes unacknowledged" 0, "min lag" 0, "total lag" 0, "total invalid state" 0, "total offsets pushed" 0, "total records buffered" 0, "total complete" 0, "total offsets processed" 0 }, "paused" 0, "bytes buffered" 0, "complete" 0, "workers" 32, "lat version" "3 0 0" }, "timestamp" 1642497956, "status" 200 } individual metrics such as lag , offsets buffered , pushes errors , etc are defined within the docid\ ypuz96vit27bjdiloinpb section however, metrics specific only to the pipeline endpoint are defined as follows true left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type related links docid\ zjlpeecj74 lp0ciflqh0 docid k bmmfc36rfofhny2fw2