LAT Sink Configuration

data pipelines are now the preferred method for loading data into the ocient system for details, see docid\ xq0tg7yph vn62uwufibu the sink configuration controls the destination of data in the lat pipeline sink a sink configuration object required keys /#sinktype sink type type of sink to use in the pipeline type string required yes default allowed values ocient see /#ocient sink for additional configuration file see /#file sink for additional configuration ocient sink the {{ocient}} sink allows lat to connect to an ocient cluster to write rows to one or more tables required keys /#sinkremotes sink remotes array of one or more ocient loader nodes, in host\ port, format type string\[] required yes default sink batch records number of records to buffer per partition before flushing records to ocient type int required no default 1000 sink batch duration time based flushing parameter, in milliseconds records will flush to ocient after this duration has elapsed with no new activity, even if fewer than batch records records have been processed type int required yes default 30000 sink idle partition polling period time based polling parameter, in milliseconds this sink will periodically poll the remote for progress on write durability for idle partitions type int required no default 60000 sink request timeout request timeout when communicating with ocient remotes, in milliseconds type int required no default 300000 sink request backoff duration to delay after a failed request to an ocient remote prior retrying, in milliseconds type int required no default 1000 sink request jitter additional duration to delay after a failed request to an ocient remote prior retrying, in milliseconds the total delay incurred prior to a given retry is request backoff + rand(0, request jitter) type int required no default 5000 sink high watermark high watermark memory point, in bytes the lat will stop pushing new rows to memory buffers it will not resume pushing rows into the memory buffers until low watermark is reached type int required no default 1000000000 sink low watermark low watermark memory point, in bytes after reaching high watermark , the lat will begin pushing rows to memory buffers again when this memory level is reached type int required no default 500000000 sink storage scope id uuid of the storage scope that rows will be associated with the scope with the given uuid must already exist in the target cluster type string required no default null sink skip page replication a boolean value to determine whether to omit page replicas for the specified storage scope this is ignored if sink storage scope id is not specified or has already been seen by the remotes type boolean required no default false sink netty event loop group threads the number of threads in the netty event loop group used to communicate with remotes type int required no default 1 example ocient sink configuration { "sink" { "type" "ocient", "remotes" \["loader0 5050", "loader1 5050", "loader2 5050"] } } file sink a sink type for testing lat pipelines that writes the transformed data to local https //jsonlines org/ files required keys /#sinklocation sink location an absolute or relative path to the location that the sink should write files to type string required yes default example file sink configuration { "sink" { "type" "file", "location" "out/" } } external sink configuration rather than including a sink directly within the pipeline, it is also possible to configure a pipeline to use a sink that is specified externally sinks can be managed (created, deleted, and more) using the docid\ xpvlz0ewuxmgynxvxz jb a sink must exist before a pipeline can use it there are three ways to configure a pipeline to use a sink if a sink is included directly within a pipeline (using the docid\ n1rdxyeldzyhmqwsd0wyz ), it will be used if a sink is not specified within the pipeline, you can specify a docid\ uiqywt8ec9gszunelylqr that corresponds to a sink previously created using the lat client if neither sink nor sink name is specified in a pipeline, the default sink will be used if a default sink has not been created using the lat client, a pipeline must specify either a sink or a sink name related links docid\ tt6tfoulap0mt aycm2ka