Ocient Simulator

{{ocient}} provides a simulated database system upon request for development or local workloads with no performance requirements this simulator system is a lightweight package that emulates a complete ocient system with these features sql port for connection using any of the supported connectors from ocient no nvme drive or {{linux}} huge page requirements to request access to an ocient simulator, please contact ocient support prerequisites the ocient simulator has these hardware and software requirements hardware 8gb system ram 64 bit {{intel}} architecture cpu (x86) allocated disk storage equal to twice the size of the data to load if you run the ocient simulator on a virtual machine, you must configure your virtual machine cpu to support avx2 instructions by setting the cpu type to host software {{docker}} engine version 20 10 10 or greater with docker compose version 1 9 0 or greater if you do not have the minimum version of docker engine installed, you can use this workaround to run the ocient simulator with an older version add the yaml option security opt \["seccomp\ unconfined"] to the lat service in the docker compose yml file in between the image and hostname keys lat image "ocient/lat/simulator 3 0 2" security opt \["seccomp\ unconfined"] hostname "lat" add the command line argument security opt seccomp=unconfined immediately after the docker run command in both jdbc sh and lat client sh scripts exec docker run security opt seccomp=unconfined it rm v $(pwd) /tmp/ocient w /tmp/ocient network "$network" ocient/jdbc 2 107exec docker run security opt seccomp=unconfined $env arg rm v $(pwd) /tmp/ocient w /tmp/ocient network "$network" ocient/lat client 3 0 2 "$@" install the ocient simulator ocient distributes its simulator as the tarball simulator archive tar upon receiving the file, you can extract it and load the container images into the docker daemon with the provided load all images sh file some file names might differ based on the simulator release version from a terminal window, create a new empty directory mkdir simulator workspace and then change the directory to the new one cd simulator workspace copy or move the tarball file into the new directory extract the file tar xf simulator archive tar the simulator package contains these files file name description configure sql the system configuration script that bootstraps the simulator docker compose yml the docker compose file that defines the containers that constitute the simulator system lat client tar , lat simulator tar , ocient jdbc image tar , rolehostd simulator tar these docker images comprise various components of the ocient system jdbc sh the jdbc cli launch script for connection to the sql interface of the simulator lat client sh the loading and transformation (lat) client launch script for connection to the lat of the simulator load all images sh this script loads all the tarball files into the docker daemon load the ocient system tarball files that form the simulator /load all images sh if you do not have access privileges to the docker daemon, you might receive the permission denied error when you run docker commands use sudo or add your user to the docker group this output shows an example of a successful load operation the exact output of your simulator might differ depending on what version you download a0c3d7b00a1e loading layer \[=====================================>] 59 26mb/59 26mb 5d75db409355 loading layer \[=====================================>] 329 6mb/329 6mb b4f0ab1562a2 loading layer \[=====================================>] 3 072kb/3 072kb 6b6be244d498 loading layer \[=====================================>] 94 76mb/94 76mb loaded image ocient/lat/simulator 2 3 0 fb4617b63504 loading layer \[=====================================>] 29 39mb/29 39mb f5745a52e122 loading layer \[=====================================>] 4 608kb/4 608kb d2d33759728d loading layer \[=====================================>] 12mb/12mb f6533a5b289e loading layer \[=====================================>] 10 24kb/10 24kb a3f69486f426 loading layer \[=====================================>] 23 64mb/23 64mb 84ff92691f90 loading layer \[=====================================>] 10 24kb/10 24kb loaded image ocient/lat client 2 3 0 loaded image ocient/jdbc 2 100 475a54c2a93d loading layer \[=====================================>] 65 52mb/65 52mb 314fd346d653 loading layer \[=====================================>] 1 32gb/1 32gb loaded image ocient/rolehostd/simulator 23 0 0 1680124029 start and configure the cluster each ocient simulator is a series of nodes that run in their own individual docker containers the docker compose command enables networking, general administration, and orchestration of these nodes the commands you use based on this section might differ with your docker version these steps use the docker compose command for the simulator setup if you are using an older version of docker with docker compose (a dash instead of a space), use that instead start the cluster using the docker compose command docker compose up d the cluster startup displays output that is similar to this output \[+] running 7/7 ⠿ network simulator workspace ocientdbnet created 0 1s ⠿ container simulator workspace sql 1 started 20 2s ⠿ container simulator workspace foundation0 1 started 15 4s ⠿ container simulator workspace foundation2 1 started 17 4s ⠿ container simulator workspace foundation1 1 started 16 4s ⠿ container simulator workspace loader 1 started 14 4s ⠿ container simulator workspace lat 1 started 7 1s ensure that you run every command from the same directory where you extracted the simulator archive tar gz archive using the bundled jdbc cli, connect to the cluster and run the contents of the configure sql file that is distributed with the simulator the jdbc cli is bundled with the simulator as the jdbc sh script run this script with no arguments after starting the cluster /jdbc sh after loading the jdbc cli, the system prompts you to enter a username and password the default simulator username and password are admin\@system and admin you can also pass these credentials later in a connect statement, for example connect to jdbc\ ocient //sql 4050/system user admin\@system using admin; the interface switches to the jdbc cli interactive mode, as indicated by the ocient> prompt run the configuration file jdbc cli source configure sql; the configuration displays output that is similar to this output connected to jdbc\ ocient //sql 4050/system modified 0 rows modified 0 rows modified 0 rows warning node must be restarted for changes to take effect (the streamloader role has been added to the configuration, but the streamloader is not yet active on the node to have this change take effect, a restart of the streamloader and foundation nodes is required) modified 0 rows modified 0 rows modified 0 rows after completing its modifications, the simulator warns that it needs to restart for the changes to take effect quit the interactive mode jdbc cli quit restart the cluster docker compose restart after the nodes restart, verify that the simulator has started and the test database is accessible by connecting to it with the jdbc cli launch the jdbc connector again /jdbc sh connect to the test database in the jdbc cli jdbc cli connect to jdbc\ ocient //sql 4050/test; the system displays a message to confirm a successful connection jdbc cli connected to jdbc\ ocient //sql 4050/test load data with the lat client the simulator cluster includes an lat node, which can load data from external sources, such as an {{aws}} s3 bucket this section has the steps to create a basic table and pipeline with the lat client, which you can then use to load data from a sample data set into a simulator database if you are not already in the jdbc cli interactive mode, start it and connect to the test database using the steps in ocient simulator docid\ dvu6y5harufaah02imhjd create a table to load from the jdbc cli, enter this sql create table statement create table loading data type coverage ( col bigint bigint null, col binary binary(2) null, col boolean bool null, col char char(4) null, col date date null, col decimal decimal(18,4) null, col double double null, col float float null, col int int null, col int array int\[] null, col ipv4 ipv4 null, col ip ip null, col smallint smallint null, col point point null, col linestring linestring null, col polygon polygon null, col time time null, col timestamp timestamp null, col tinyint tinyint null, col tuple tuple<\<int, varchar(255)>> null, col uuid uuid null, col varbinary varbinary(255) null, col varchar varchar(255) null ); create a pipeline to load the data using the lat client exit the jdbc cli by entering the quit command create a new file named pipeline json in the directory where you are running the simulator copy this json text and paste it into the pipeline json file { "version" 2, "workers" 1, "source" { "compression" "gzip", "type" "s3", &#x9; "endpoint" "https //s3 us east 1 amazonaws com", &#x9; "bucket" "ocient examples", &#x9; "file groups" { &#x9; "test" { &#x9; "prefix" "all data types/small", &#x9; "file matcher syntax" "glob", &#x9; "file matcher pattern" " output csv gz", &#x9; "sort type" "lexicographic" &#x9; } } }, "sink" { "type" "ocient", "remotes" \["loader 5050"] }, "extract" { "test" { "record type" "delimited", "headers" \[ "bigint", "binary", "boolean", "char", "date", "decimal", "double", "float", "int", "int array", "ipv4", "ip", "smallint", "point", "linestring", "polygon", "time", "timestamp", "tinyint", "tuple", "uuid", "varbinary", "varchar" ], "record delimiter" "\n", "field delimiter" ",", "field optionally enclosed by" "\\"", "num header lines" 1 } }, "transform" { "file groups" { "test" { "tables" { "test loading data type coverage" { "columns" { "col bigint" "bigint", "col binary" "binary", "col boolean" "boolean", "col char" "char", "col date" "date", "col decimal" "decimal", "col double" "to number(double)", "col float" "to number(float)", "col int" "int", "col int array" "parse array(int array, '\[', ',', `1`)", "col ipv4" "ipv4", "col ip" "ip", "col smallint" "smallint", "col point" "point", "col linestring" "linestring", "col polygon" "polygon", "col time" "time", "col timestamp" "replace(replace(timestamp, 't', ' '), 'z', '')", "col tinyint" "tinyint", "col tuple" "to tuple(tuple, '(', ',')", "col uuid" "uuid", "col varbinary" "varbinary", "col varchar" "varchar" } } } } } } } execute this command to create the pipeline by using the lat client script /lat client sh pipeline create pipeline pipeline json unlike the jdbc cli, the lat client is not interactive you must issue each command for the lat client as a separate /lat client command start the pipeline to load the data /lat client sh pipeline start the sample data set can take about 30 seconds to fully load inspect the tables using the jdbc cli to validate the load of the data set launch the jdbc cli /jdbc sh connect to the test database jdbc cli connect to jdbc\ ocient //sql 4050/test; verify the loading data type coverage table is present jdbc cli list tables; verify the row count in the table using this sql statement select count( ) from loading data type coverage; if the load operation is complete, the output table shows a count of 100,000 rows count( ) \ 100000 fetched 1 row when you are confident your data is fully loaded, you can dismantle your pipeline stop the pipeline /lat client sh pipeline stop delete the pipeline /lat client sh pipeline delete now that your ocient simulator is loaded with data, you can add integrations for user interfaces, visualizations, and other applications for details, see ocient integrations docid\ slds1ecb2mcriy 4z 6my container configuration this section explains the configuration setup and options for the ocient simulator ports the container exposes additional ports that can be useful depending on the use case for the system exposed port description 4050 the jdbc port of the cluster 8080 ocient lat service when you load new data, the lat client connects to this port 9090 and 9091 the rest interfaces for the sql and foundation (lts) node processes, respectively environment variables you can set these environment variables to affect certain aspects of the ocient simulator environment variable description lat memory and lat memory direct configures the memory used by lat when it is running for details about these environment variables, see the lat service configuration docid\ qsaetjyytgqsqtwjwmvap page the lat node is the only node in the cluster affected by this variable example services lat environment lat memory '32g' resource limits (cpu, memory, disk usage) docker supports a variety of options to limit container access to host resources refer to the docker runtime options with memory, cpus, and gpus page for the full list of options this section explains how constraints on specific host resources affect the ocient simulator in all of these code examples, replace image with the filename of your docker image the ocient system applies docker resource configuration changes individually on each of the nodes in the docker compose yml file (such as sql , foundation , loader , lat ) cpu cpu can be safely limited without adverse effects most workloads benefit from having multiple cpu cores available (four or more) for example, to add cpu limits to the sql node, add this text to the docker compose yml file in the simulator package replace the '4' with the cpu limit that you want services sql image image deploy resources limits cpus '4' memory (ram) the ocient simulator requires sufficient ram to execute all workloads stability or uptime of the system is not guaranteed in low or out of memory conditions if you need to limit the memory of the running docker container, it is recommended to benchmark the container memory usage on your workload and provide at least 1gb of ram above the maximum observed when limiting memory of the ocient simulator, do not use the oom kill disable option, otherwise the database can become unresponsive for a significant period of time after running out of memory to add memory limits to the sql node, add this text to the docker compose yml file in the simulator package replace the 8g with the memory limit that you want services sql image image deploy resources limits memory 8g disk usage when running the simulator without mounted data, you can specify the size in bytes of the data files created the data files must be large enough to fit any data sets that you load the file size defaults to 2gib to specify a different file size, you can configure the rolehostd startup command in the docker compose yml file add the command option loopbackdevicesize with a new value to configure a new file size in this example, the file size on the foundation0 node is configured to 16gib services sql image image command " bootstrap '{adminhost \\"sql\\", ipaddress \\"foundation0\\", loopbackdevicesize \\"16gib\\"}}'" if you want to specify a new file size value, you must do it before you create your cluster while you cannot resize files on a running cluster, you can change this value by deleting and recreating your cluster this process is irreversible and erases any existing databases and tables to resize file on an existing cluster stop your cluster by running docker compose down clear the volumes directory by running rm rf volumes/ if you have not yet edited the docker compose yml file, edit the file to include new loopbackdevicesize values restart your cluster by running docker compose up d simulator logging the ocient simulator sends all logs to stdout and stderr as well as any logging driver configured for the host docker installation or the container this logging allows for easy inspection of container logs using the docker logs https //docs docker com/engine/reference/commandline/logs/ command for details on logging with docker, consult the logging section in the docker documentation setting logging levels you can modify the logging level for each node in the cluster by configuring the rolehostd startup command in the docker compose yml file add the command option loglevel with a new log level supported values include error , info , warn , verbose , or debug defaults to info for details, see log monitoring docid\ goypkivlud77sz0sx ekq example services sql image image command " bootstrap '{adminhost \\"sql\\", ipaddress \\"foundation0\\", loglevel \\"debug\\"}}'" other logging configurations you can customize your logging configuration for each node by making changes to the docker compose yml file in this example, the sql node has special configurations for its max size and max file values, meaning the container should have no more than five log files of 10mb in size example services sql image image logging driver json file options max size 10m max file 5 use docker logs to retrieve the logs in this example, the tail option limits the output to include only the last 10 lines of logs for more options, see the docker compose logs page docker logs tail 10 container related links commands supported by the ocient jdbc cli program docid\ fwb2ppe2b5li l pisj25 database, tables, views, and indexes docid\ ejutg6wjnk5eg55kizq8d {{linux}} is the registered trademark of linus torvalds in the u s and other countries