Skip to main content
Data Pipelines are now the preferred method for loading data into the System. For details, see Load Data.
The Ocient Loading and Transformation (LAT) is a service used for transforming and streaming data into Ocient. The LAT transforms records from a configured Source using the JMESPath transformation language and streams records into Ocient tables.

​
LAT Benefits

  • Exactly-once processing from a Source
  • Selecting data and arrays from nested JSON data
  • Transforming records and routing into one or more tables
  • Scalar data transformation (e.g., concatenate strings, uppercasing, rounding)
  • Dynamic schema change handling

​
LAT Data Flow

The LAT consumes JSON records, transforms them, and loads them into one or more Ocient tables. This process is referred to as a “pipeline”, with a single pipeline running per LAT instance. The pipeline is described by a JSON-based configuration that instructs the LAT how to consume records from a Source, how to transform the records, and which Ocient table to route the records to. As records are consumed, the transformations described in the configuration are applied to each record before forwarding the resulting records to Ocient. Each pipeline configuration can map a record to 1 or more Ocient Tables, each table with its own transformation configuration. LAT can optionally be configured to write records which were unable to be processed to an error topic. Multiple LAT instances can be used in tandem to further parallelize a pipeline. The LAT makes use of the Kafka Consumer library to ensure that each Kafka Partition is only assigned to a single instance. To use multiple instances, configure each LAT instance with the same pipeline configuration. The LAT instances will automatically coordinate and distribute the load across the instances. The LAT is controlled by an HTTP endpoint and a command line interface (CLI) is supplied to perform common actions such as starting, stopping, or updating the LAT pipelines. Connect to Ocient Ingest Data with Legacy LAT Reference At the Whiteboard with Ocient: Loading and Transformation
Last modified on May 27, 2026