LAT Client Command Line Interface

data pipelines are now the preferred method for loading data into the ocient system for details, see load data docid 1nw163anqyhk58cqmepjs the lat client can be used to interact with a running lat instance it supports subcommands for interacting with pipelines, and previewing transformations the lat client is distributed in the form of a {{python}} wheel file contact {{ocient}} support for the wheel that corresponds to the lat version prerequisites python >= 3 8 pip3 >= 20 2 3 if pip install fails, try upgrading pip wheel python package pip install wheel install it is recommended to install the wheel in a python virtual environment to avoid conflicts with globally installed python packages for the install command, replace $version with the latest version of the lat client steps create the virtual environment python3 m venv venv activate the virtual environment source venv/bin/activate install the wheel pip install lat client $version py3 none any whl run commands with lat client command args when a new terminal is opened, repeat step 2 to activate the environment and gain access to the lat client command usage get help on the command line lat client help lat client \<subcommand> help common arguments some arguments are available on all subcommands for convenience, most of them can also be set using an environment variable \ no verify skip certificate validation when connecting to lat ignored if using http scheme example no verify \ hosts (lat hosts) one or more lat hosts to orchestrate valid domain names or ip addresses can be used example hosts http //192 168 0 1 8080 http //192 168 0 2 8081 environment export lat hosts="http //10 4 0 1 8080,http //10 4 0 2 8081" \ oauth domain (lat oauth domain) {{okta}} oauth domain to use for token acquisition example oauth domain https //dev 12345678 okta com environment export lat oauth domain="https //dev 12345678 okta com" \ oauth server (lat oauth server) okta oauth authorization server to use for token acquisition example oauth server abcdef000ghijklm111 environment export lat oauth server="abcdef000ghijklm111" \ client id (lat client id) okta client id to use for token acquisition example client id 12345678 environment export lat client id="12345678" \ client secret (lat client secret) okta client secret to use for token acquisition example client secret abc123 environment export lat client secret="abc123" \ oauth http proxy (lat oauth http proxy) http proxy url to use for token acquisition authentication credentials can be passed in proxy url example oauth http proxy http //user\ pass\@some proxy com environment export lat oauth http proxy="http //user\ pass\@some proxy com" subcommands pipeline create create a new pipeline for most use cases, it is advisable to leave the pipeline id unset when creating a pipeline the client will set it to a random uuid to prevent deduplication across different pipelines in the event that you want deduplication between pipelines, the pipeline id should be copied from the previous pipeline and included in the new pipeline the lat transform configuration docid 4 64z9kaveebxvnxk36vb must also be the same to ensure deduplication when the client is used to create a pipeline with a file source, the client will make adjustments to the source configuration such that partitions are assigned evenly across nodes first, the client will get the number of workers from the pipeline if one is not set, it will use the minimum configured lat default workers instead then, it will set partitions = workers num nodes finally, it will set partitions assigned = \[workers node index, workers (node index + 1) 1] for each node arguments pipeline path to the pipeline configuration json file example lat client pipeline create pipeline /home/user/my new pipeline json pipeline get get the configuration for an existing pipeline if the configured pipelines are identical, print the pipeline, otherwise an explanation of the inconsistency will be provided for pipelines with a file source, partitions assigned is ignored when checking if pipelines are identical additionally, the client will validate that all partitions are assigned, and that no partition is assigned more than once example lat client pipeline get pipeline update update the configuration for an existing pipeline the new pipeline can only make changes to subfields in transform , except for any topic / file group names all other subfields of transform are allowed to change, including to the table and column fields for pipelines with a file source, partition assignments will be copied from the existing pipeline if the pipeline was running prior to the update, successful completion of this command will automatically restart the pipeline arguments pipeline path to the pipeline configuration json file example lat client pipeline update pipeline /home/user/my new pipeline json pipeline delete delete an existing pipeline unless the force flag is used, a pipeline must be stopped, or deletion will fail arguments force force a pipeline to delete regardless of running status skip validation delete a pipeline regardless of cluster consistency example lat client pipeline delete skip validation pipeline start start the configured pipeline prior to starting the pipeline, the pipeline start subcommand will validate that all specified hosts are configured with an identical pipeline, except for pipelines with a file source, which must have different partitions assigned such that each partition is assigned exactly once across all hosts example lat client pipeline start pipeline stop stop the configured pipeline example lat client pipeline stop pipeline status retrieve status of the pipeline the valid pipeline statuses are stopped , running , completed , and failed when the pipeline is failed , the file statuses will remain in processing arguments list files lists selected files in their sorted order for each file group, along with file statuses ( completed , processing , not started ) output is summarized to be human readable the system truncates large file lists arguments list all files lists all selected files in their sorted order for each file group, along with file statuses ( completed , processing , not started ) example lat client pipeline status pipeline errors retrieve errors that occur while the current pipeline runs arguments json output errors as lines of json rather than in the default human readable format max errors max errors an upper limit on the number of errors to retrieve (default is 100) only records only show records (not error messages or other information) only error messages only show error messages (not records or other information) no records show all information except records example lat client pipeline errors no records max errors 10 pipeline rebalance rebalances partitions evenly to all provided lat nodes this subcommand only applies to pipelines running file sources this command is meant to be used in the case of a node outage during a file load lat file loading does not support automatic partition re balancing and manual intervention is required the flow is as follows lat node goes offline lat operator rebalances the partitions from the offline node onto the online nodes using the client the operator should use the rebalance command and omit the offline node from the hosts argument lat node comes back online lat operator rebalances partitions using the client to include all nodes including the newly online node the operator should use the rebalance command and include all online nodes in the hosts argument example lat client pipeline rebalance sink create create a new sink configuration the sink configuration file for this subcommand should match the same format as the lat sink configuration docid 3wf6e0sg76xzhjuam5pn for example { "type" "ocient", "remotes" \["1 2 3 4 5050"]} arguments sink path to the sink configuration json file name name of the sink to create default set this as the default sink example lat client sink create sink /home/user/my sink config json name my sink name 1 default sink delete delete a sink configuration the sink configuration must not be part of a created or running pipeline arguments name name of the sink to delete example lat client sink delete name my sink name 1 sink list list all configured sinks example lat client sink list sink get get a sink configuration by id arguments name name of the sink to get configuration for example lat client sink get name my sink name 1 preview preview a transformation at most one of transform or pipeline can be provided if neither is provided, the host will attempt to use the transformation configured in its pipeline you can specify the extract or pipeline option if you specify none of these options, the host uses the json extractor by default for details about record and extractor types, see the lat extract configuration docid\ osvkjtvcq6ngq 6quhvbu you must specify the topic or file group option, which should match the topic or file group key in the specified transform section arguments topic name of the topic the records are associated with file group name of the file group the records are associated with records path to a file of records to transform record formats can be of delimited records (e g , csv, tsv), json records, or fixed width binary records extract \[optional] path to a json file containing the extract section of a pipeline definition to use for extraction transform \[optional] path to a json file containing the transform section of a pipeline definition to use for transformation pipeline \[optional] path to a json file containing a pipeline to use for transformation and extraction, if present example without pipeline lat client preview topic test topic records /data/my records extract /home/user/my extract json transform /home/user/my transform json example using pipeline lat client preview topic test topic records /data/my records pipeline /home/user/my pipeline json common workflows check on status of lat pipelines $ lat client pipeline status 10 4 0 1 8080 stopped 10 4 0 2 8080 stopped 10 4 0 3 8080 stopped $ lat client pipeline status hosts https //10 4 0 1 8080 10 4 0 1 8080 stopped updating an existing lat pipeline lat client pipeline update pipeline /home/user/my new pipeline json if the pipeline was running prior to the update, successful completion of this command will automatically restart the pipeline if unsuccessful, the cli will report an error with an explanation of what is wrong with the command common issues are invalid json, missing a required column an unsuccessful update of the pipeline config does not impact actively running pipelines restart the pipeline \# first, stop the running pipelines to send custom hosts, use hosts ip1\ port ip2\ port $ lat client pipeline stop stopped 10 4 0 1 8080, 10 4 0 2 8080, 10 4 0 3 8080 \# then, start the pipelines $ lat client pipeline start started 10 4 0 1 8080, 10 4 0 2 8080, 10 4 0 3 8080 check multiple lat nodes to see if the pipeline configurations are compatible \# hosts with compatible pipelines $ lat client pipeline get hosts https //10 4 0 1 8080 https //10 4 0 2 8080 https //10 4 0 3 8080 { "pipeline id" "91326229 5fc6 4542 99ce 87bdcb00a978", "version" 2, "source" { }, "sink" { }, "transform" { } } \# hosts with incompatible pipeline $ lat client pipeline get hosts https //10 4 0 1 8080 https //10 4 0 2 8080 https //10 4 0 3 8080 hosts responded with incompatible pipelines b5a6e8647cd3a28d4a5bd07cd1c491ce 10 4 0 1 8080, 10 4 0 2 8080 160f270e39d7a51a431c841613e4dc4e 10 4 0 3 8080 the cli will compare the md5 hash of the pipeline configurations on all nodes and respond that all pipelines match or are inconsistent lat client command line interface troubleshooting if the certificate authorities on the system running the python client (lat client) need to be updated, an error can occur this type of error message might appear ssl sslcertverificationerror \[ssl certificate verify failed] certificate verify failed unable to get local issuer certificate ( ssl c 997) you can run this command to resolve the error pip install upgrade certifi the root cause of this error can be a connection to either the lat server over ssl or an authentication to okta to obtain an access token over ssl you can run the same command in both cases to resolve the issue when the ssl certificate is self signed on the lat server, you can use the no verify flag when you connect to the lat server without verifying the ssl certificate related links ingest data with legacy lat reference docid ul5nilgyd7y wjccg59d install an ocient system docid\ qhvyfg5eqo7rz8ssuvk1p