LAT Overview

data pipelines are now the preferred method for loading data into the ocient system for details, see docid\ xq0tg7yph vn62uwufibu the {{ocient}} loading and transformation (lat) is a service used for transforming and streaming data into ocient the lat transforms records from a configured source using the https //jmespath org/ transformation language and streams records into ocient tables lat benefits exactly once processing from a source selecting data and arrays from nested json data transforming records and routing into one or more tables scalar data transformation (e g , concatenate strings, uppercasing, rounding) dynamic schema change handling lat data flow the lat consumes json records, transforms them, and loads them into one or more ocient tables this process is referred to as a "pipeline", with a single pipeline running per lat instance the pipeline is described by a json based configuration that instructs the lat how to consume records from a source, how to transform the records, and which ocient table to route the records to as records are consumed, the transformations described in the configuration are applied to each record before forwarding the resulting records to ocient each pipeline configuration can map a record to 1 or more ocient tables, each table with its own transformation configuration lat can optionally be configured to write records which were unable to be processed to an {{kafka}} error topic multiple lat instances can be used in tandem to further parallelize a pipeline the lat makes use of the kafka consumer library to ensure that each kafka partition is only assigned to a single instance to use multiple instances, configure each lat instance with the same pipeline configuration the lat instances will automatically coordinate and distribute the load across the instances the lat is controlled by an http endpoint and a command line interface (cli) is supplied to perform common actions such as starting, stopping, or updating the lat pipelines related links docid\ y ge7svqdmn 9yie2iuv6 docid\ tt6tfoulap0mt aycm2ka related links https //www youtube com/watch?v=d3exgc20rec