Ocient Simulator

 provides a simulated database system upon request for development or local workloads with no performance requirements.

This simulator system is a lightweight package that emulates a complete Ocient system with these features:

  • SQL port for connection using any of the supported connectors from Ocient.
  • No NVMe drive or huge page requirements.

To request access to an Ocient simulator, please contact Ocient Support.

Prerequisites

The Ocient simulator has these hardware and software requirements:

Hardware

  • 8GB system RAM
  • 64-bit architecture CPU (x86)
  • Allocated disk storage equal to twice the size of the data to load.

Software

 Engine version 20.10.10 or greater with Docker Compose version 1.9.0 or greater.

If you do not have the minimum version of Docker Engine installed, you can use this workaround to run the Ocient Simulator with an older version.

1

Add the YAML option security_opt: ["seccomp:unconfined"] to the LAT service in the docker-compose.yml file in between the image and hostname keys.

YAML

2

Add the command-line argument --security-opt seccomp=unconfined immediately after the docker run command in both jdbc.sh and lat_client.sh scripts.

Shell

Shell


Install the Ocient Simulator

Ocient distributes its simulator in compressed tarball format (.tar.gz). Upon receiving the file, you can decompress and load it into the Docker daemon with the docker load command.

Some file names might differ based on the simulator release version.

1

From a terminal window, create a new empty directory.

Shell


And then change the directory to the new one.

Shell

2

Copy or move the compressed tarball file into the new directory.

3

Decompress the file.

Shell


The simulator package contains the following files.

File Name

Description

configure.sql

The system configuration script that bootstraps the simulator.

docker-compose.yml

The Docker Compose file that defines the containers that constitute the simulator system.

ocient_simulator_images.tar

A compressed repository of Docker images referenced by the docker-compose.yml file that includes:

- rolehostd

- LAT

- JDBC CLI

- LAT client

jdbc.sh

The JDBC CLI launch script for connection to the SQL interface of the simulator.

lat_client.sh

The Loading and Transformation (LAT) client launch script for connection to the LAT of the simulator.

4

Load the ocient_simulator_images.tar file that contains the individual images that form the simulator.

Shell


If you do not have access privileges to the Docker daemon, you might receive a “Permission denied” error when you run docker commands. Use sudo docker or add your user to the Docker group.

This output shows an example of a successful load operation.

Shell


Start and Configure the Cluster

Each Ocient simulator is a series of nodes that run in their own individual Docker containers. The docker compose command enables networking, general administration, and orchestration of these nodes.

The commands you use based on this section might differ with your Docker version.

These steps use the docker compose command for the simulator setup. If you are using an older version of Docker with docker-compose (a dash instead of a space), use that instead.

1

Start the cluster using the docker compose command.

Shell


The cluster startup displays output that is similar to this output.

Shell


Ensure that you run every command from the same directory where you extracted the simulator_archive.tar.gz archive.

2

Using the bundled JDBC CLI, connect to the cluster and run the contents of the configure.sql file that is distributed with the simulator.

The JDBC CLI is bundled with the simulator as the jdbc.sh script. Run this script with no arguments after starting the cluster.

Shell


After loading the JDBC CLI, the system prompts you to enter a username and password. The default simulator username and password are admin@system and admin.

You can also pass these credentials later in a CONNECT statement, for example:

CONNECT TO jdbc:ocient://sql:4050/system USER admin@system USING admin;

The interface switches to the JDBC CLI interactive mode, as indicated by the Ocient> prompt. Run the configuration file.

JDBC CLI


The configuration displays output that is similar to this output.

Shell

3

After completing its modifications, the simulator warns that it needs to restart for the changes to take effect. Quit the interactive mode.

JDBC CLI


Restart the cluster.

Shell

4

After the nodes restart, verify that the simulator has started and the test database is accessible by connecting to it with the JDBC CLI.

Launch the JDBC connector again.

Shell


Connect to the test database in the JDBC CLI.

JDBC CLI


The system displays a message to confirm a successful connection.

JDBC CLI


Load Data with the LAT Client

The simulator cluster includes an LAT node, which can load data from external sources, such as an S3 bucket. This section has the steps to create a basic table and pipeline with the LAT client, which you can then use to load data from a sample data set into a simulator database.

If you are not already in the JDBC CLI interactive mode, start it and connect to the test database as demonstrated in the prior section.

1

Create a table to load. From the JDBC CLI, enter this SQL CREATE TABLE statement.

SQL

2

Create a pipeline to load the data using the LAT client. Exit the JDBC CLI by entering the quit command.

Create a new file named pipeline.json in the directory where you are running the simulator.

Copy this JSON text and paste it into the pipeline.json file.

JSON

3

Execute this command to create the pipeline by using the LAT client script.

Shell


Unlike the JDBC CLI, the LAT client is not interactive. You must issue each command for the LAT client as a separate ./lat_client command.

4

Start the pipeline to load the data.

Shell


The sample data set can take about 30 seconds to fully load.

Inspect the tables using the JDBC CLI to validate the load of the data set. Launch the JDBC CLI.

Shell


Connect to the test database.

JDBC CLI


Verify the loading.data_type_coverage table is present.

JDBC CLI


Verify the row count in the table using this SQL command.

SQL


If the load operation is complete, the output table shows a count of 100,000 rows.

SQL

5

When you are confident your data is fully loaded, you can dismantle your pipeline.

Stop the pipeline.

Shell


Delete the pipeline.

Shell




Container Configuration

This section explains the configuration setup and options for the Ocient simulator.

Ports

The container exposes additional ports that can be useful depending on the use case for the system.

Exposed Port

Description

4050

The JDBC port of the cluster.

8080

Ocient LAT service. When you load new data, the LAT client connects to this port.

9090 and 9091

The REST interfaces for the SQL and Foundation (LTS) node processes, respectively.

Environment Variables

You can set these environment variables to affect certain aspects of the Ocient simulator.

Environment variable

Description

LAT_MEMORY and LAT_MEMORY_DIRECT

Configures the memory used by LAT when it is running. For details about these environment variables, see the Service Configuration page.

The LAT node is the only node in the cluster that is affected by this variable.

Example

YAML


Resource Limits (CPU, Memory, Disk Usage)

Docker supports a variety of options to limit container access to host resources. Refer to the Docker Runtime options with Memory, CPUs, and GPUs page for the full list of options. This section explains how constraints on specific host resources affect the Ocient Simulator.

In all of these code examples, replace IMAGE with the file name of your Docker image.

The Ocient system applies Docker resource configuration changes individually on each of the nodes in the docker-compose.yml file (such as sql, foundation, loader, lat).

CPU

CPU can be safely limited without adverse effects. Most workloads benefit from having multiple CPU cores available (four or more).

For example, to add CPU limits to the sql node, add this text to the docker-compose.yml file in the simulator package. Replace the '4' with the CPU limit that you want.

YAML


Memory (RAM)

The Ocient simulator requires sufficient RAM to execute all workloads. Stability or uptime of the system is not guaranteed in low or out-of-memory conditions. If you need to limit the memory of the running Docker container, it is recommended to benchmark the container memory usage on your workload and provide at least 1GB of RAM above the maximum observed.

When limiting memory of the Ocient simulator, do not use the --oom-kill-disable option, otherwise the database can become unresponsive for a significant period of time after running out of memory.

To add memory limits to the sql node, add this text to the docker-compose.yml file in the simulator package. Replace the 8g with the memory limit that you want.

YAML


Disk Usage

When running the simulator without mounted data, you can specify the size in bytes of the data files created. The data files must be large enough to fit any data sets that you load. The file size defaults to 2GiB.

To specify a different file size, you can configure the rolehostd startup command in the docker-compose.yml file. Add the command option loopbackDeviceSize with a new value to configure a new file size.

In this example, the file size on the foundation0 node is configured to 16GiB.

YAML


If you want to specify a new file size value, you must do it before you create your cluster.

While you cannot resize files on a running cluster, you can change this value by deleting and recreating your cluster. This process is irreversible and erases any existing databases and tables.

To resize file on an existing cluster:

  1. Stop your cluster by running docker compose down.
  2. Clear the volumes directory by running rm -rf volumes/.
  3. If you have not yet edited the docker-compose.yml file, edit the file to include new loopbackDeviceSize values.
  4. Restart your cluster by running docker compose up -d.

Logging

The Ocient simulator sends all logs to stdout and stderr as well as any logging driver configured for the host Docker installation or the container. This logging allows for easy inspection of container logs via the docker logs command.

For more information on logging with Docker, consult the logging section in the Docker documentation.

Setting Logging Levels

You can modify the logging level for each node in the cluster by configuring the rolehostd startup command in the docker-compose.yml file. Add the command option logLevel with a new log level.

Supported values include error, info, warn, verbose, or debug. Defaults to info.

For more information, see Error Monitoring.

Example

YAML


Other Logging Configurations

You can customize your logging configuration for each node by making changes to the docker-compose.yml file. In this example, the sql node has special configurations for its max-size and max-file values, meaning the container should have no more than five log files of 10MB in size.

Example

YAML


Use docker logs to retrieve the logs. In this example, the --tail option limits the output to include only the last 10 lines of logs.

For more options, see the docker compose logs page.

Shell



 is the registered trademark of Linus Torvalds in the U.S. and other countries.