LAT Sink Configuration

data pipelines are now the preferred method for loading data into the ocient system for details, see load data docid\ qxg07ea5hv1vxat6neyg the sink configuration controls the destination of data in the lat pipeline sink a sink configuration object required keys lat sink configuration docid\ zxx 34ywq1kvksqemnyyy sink type type of sink to use in the pipeline type string required yes default allowed values ocient see lat sink configuration docid\ zxx 34ywq1kvksqemnyyy for additional configuration file see lat sink configuration docid\ zxx 34ywq1kvksqemnyyy for additional configuration ocient sink the {{ocient}} sink allows lat to connect to an ocient cluster to write rows to one or more tables required keys lat sink configuration docid\ zxx 34ywq1kvksqemnyyy sink remotes array of one or more ocient loader nodes, in host\ port, format type string\[] required yes default sink batch records number of records to buffer per partition before flushing records to ocient type int required no default 1000 sink batch duration time based flushing parameter, in milliseconds records will flush to ocient after this duration has elapsed with no new activity, even if fewer than batch records records have been processed type int required yes default 30000 sink idle partition polling period time based polling parameter, in milliseconds this sink will periodically poll the remote for progress on write durability for idle partitions type int required no default 60000 sink request timeout request timeout when communicating with ocient remotes, in milliseconds type int required no default 300000 sink request backoff duration to delay after a failed request to an ocient remote prior retrying, in milliseconds type int required no default 1000 sink request jitter additional duration to delay after a failed request to an ocient remote prior retrying, in milliseconds the total delay incurred prior to a given retry is request backoff + rand(0, request jitter) type int required no default 5000 sink high watermark high watermark memory point, in bytes the lat will stop pushing new rows to memory buffers it will not resume pushing rows into the memory buffers until low watermark is reached type int required no default 1000000000 sink low watermark low watermark memory point, in bytes after reaching high watermark , the lat will begin pushing rows to memory buffers again when this memory level is reached type int required no default 500000000 sink storage scope id uuid of the storage scope that rows will be associated with the scope with the given uuid must already exist in the target cluster type string required no default null sink skip page replication a boolean value to determine whether to omit page replicas for the specified storage scope this is ignored if sink storage scope id is not specified or has already been seen by the remotes type boolean required no default false sink netty event loop group threads the number of threads in the netty event loop group used to communicate with remotes type int required no default 1 example ocient sink configuration { "sink" { "type" "ocient", "remotes" \["loader0 5050", "loader1 5050", "loader2 5050"] } } file sink a sink type for testing lat pipelines that writes the transformed data to local jsonl https //jsonlines org/ files required keys lat sink configuration docid\ zxx 34ywq1kvksqemnyyy sink location an absolute or relative path to the location that the sink should write files to type string required yes default example file sink configuration { "sink" { "type" "file", "location" "out/" } } external sink configuration rather than including a sink directly within the pipeline, it is also possible to configure a pipeline to use a sink that is specified externally sinks can be managed (created, deleted, and more) using the lat client command line interface docid\ h7orjtikzcqpw0 ilkvsl a sink must exist before a pipeline can use it there are three ways to configure a pipeline to use a sink if a sink is included directly within a pipeline (using the lat sink configuration docid\ zxx 34ywq1kvksqemnyyy ), it will be used if a sink is not specified within the pipeline, you can specify a lat pipeline configuration docid\ aczjrpa6a8wlrrdqa swc that corresponds to a sink previously created using the lat client if neither sink nor sink name is specified in a pipeline, the default sink will be used if a default sink has not been created using the lat client, a pipeline must specify either a sink or a sink name related links ingest data with legacy lat reference docid\ luijhab6vyj6g1gn5bhth