LAT Sink Configuration

data pipelines are now the preferred method for loading data into the ocient system for details, see docid\ zncvnrhsf6fg1yvqk6mxt the sink configuration controls the destination of data in the lat pipeline sink a sink configuration object required keys docid\ tqopqz139pn8ti995a31m sink type type of sink to use in the pipeline left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type allowed values ocient see docid\ tqopqz139pn8ti995a31m for additional configuration file see docid\ tqopqz139pn8ti995a31m for additional configuration ocient sink the {{ocient}} sink allows lat to connect to an ocient cluster to write rows to one or more tables required keys docid\ tqopqz139pn8ti995a31m sink remotes array of one or more ocient loader nodes, in host\ port, format left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink batch records number of records to buffer per partition before flushing records to ocient left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink batch duration time based flushing parameter, in milliseconds records will flush to ocient after this duration has elapsed with no new activity, even if fewer than batch records records have been processed left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink idle partition polling period time based polling parameter, in milliseconds this sink will periodically poll the remote for progress on write durability for idle partitions left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink request timeout request timeout when communicating with ocient remotes, in milliseconds left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink request backoff duration to delay after a failed request to an ocient remote prior retrying, in milliseconds left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink request jitter additional duration to delay after a failed request to an ocient remote prior retrying, in milliseconds the total delay incurred prior to a given retry is request backoff + rand(0, request jitter) left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink high watermark high watermark memory point, in bytes the lat will stop pushing new rows to memory buffers it will not resume pushing rows into the memory buffers until low watermark is reached left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink low watermark low watermark memory point, in bytes after reaching high watermark , the lat will begin pushing rows to memory buffers again when this memory level is reached 0 left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink storage scope id uuid of the storage scope that rows will be associated with the scope with the given uuid must already exist in the target cluster left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink skip page replication a boolean value to determine whether to omit page replicas for the specified storage scope this is ignored if sink storage scope id is not specified or has already been seen by the remotes left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type sink netty event loop group threads the number of threads in the netty event loop group used to communicate with remotes left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type example ocient sink configuration { "sink" { "type" "ocient", "remotes" \["loader0 5050", "loader1 5050", "loader2 5050"] } } file sink a sink type for testing lat pipelines that writes the transformed data to local https //jsonlines org/ files required keys docid\ tqopqz139pn8ti995a31m sink location an absolute or relative path to the location that the sink should write files to left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type example file sink configuration { "sink" { "type" "file", "location" "out/" } } external sink configuration rather than including a sink directly within the pipeline, it is also possible to configure a pipeline to use a sink that is specified externally sinks can be managed (created, deleted, and more) using the docid 0zve ccarf2pd0alb622e a sink must exist before a pipeline can use it there are three ways to configure a pipeline to use a sink if a sink is included directly within a pipeline (using the docid\ tqopqz139pn8ti995a31m ), it will be used if a sink is not specified within the pipeline, you can specify a docid 37u9m tt6h6wmshsx7zd that corresponds to a sink previously created using the lat client if neither sink nor sink name is specified in a pipeline, the default sink will be used if a default sink has not been created using the lat client, a pipeline must specify either a sink or a sink name related links docid\ zjlpeecj74 lp0ciflqh0