LAT Overview

data pipelines are now the preferred method for loading data into the ocient system for details, see load data docid\ qxg07ea5hv1vxat6neyg the {{ocient}} loading and transformation (lat) is a service used for transforming and streaming data into ocient the lat transforms records from a configured source using the jmespath https //jmespath org/ transformation language and streams records into ocient tables lat benefits exactly once processing from a source selecting data and arrays from nested json data transforming records and routing into one or more tables scalar data transformation (e g , concatenate strings, uppercasing, rounding) dynamic schema change handling lat data flow the lat consumes json records, transforms them, and loads them into one or more ocient tables this process is referred to as a "pipeline", with a single pipeline running per lat instance the pipeline is described by a json based configuration that instructs the lat how to consume records from a source, how to transform the records, and which ocient table to route the records to as records are consumed, the transformations described in the configuration are applied to each record before forwarding the resulting records to ocient each pipeline configuration can map a record to 1 or more ocient tables, each table with its own transformation configuration lat can optionally be configured to write records which were unable to be processed to an {{kafka}} error topic multiple lat instances can be used in tandem to further parallelize a pipeline the lat makes use of the kafka consumer library to ensure that each kafka partition is only assigned to a single instance to use multiple instances, configure each lat instance with the same pipeline configuration the lat instances will automatically coordinate and distribute the load across the instances the lat is controlled by an http endpoint and a command line interface (cli) is supplied to perform common actions such as starting, stopping, or updating the lat pipelines related links connect to ocient docid\ lb5x4qrnpht6m9v2evhuc ingest data with legacy lat reference docid\ luijhab6vyj6g1gn5bhth related links at the whiteboard with ocient loading and transformation