System Administration
Maintenance Overview

Expand and Rebalance System

If your system reaches capacity, you can add storage by installing additional nodes and drives. This process requires rebalancing the data to distribute the query processing across all available hardware, including newly installed disks or nodes. 

This tutorial covers expanding a system with additional storage.

For information on how to replace nodes that have failed or are damaged, see Replace Nodes.

For information on replacing drives that have failed or are damaged, see Replace an NVMe Node Drive.

Add Foundation Nodes

A core component of an

 System, the Foundation Nodes store user data in Ocient and perform the bulk of query processing. By adding more Foundation Nodes, a system can support extra storage capacity as well as better performance for query processing. 

System Considerations

  • No individual cluster should have more than two petabytes of storage.
  • Each cluster should have the same number of Foundation Nodes.

Prerequisites

  • The process requires either the System Administrator role or the granted UPDATE privilege on SYSTEM. 
  • Any new Foundation Nodes should have the same OS kernel version as other nodes in the system. 
  • Any new Foundation Nodes should have the same Ocient System version as the other nodes. For more information, see Ocient Application Installation.
  • Ensure that your system meets the requirements outlined in Ocient System Bootstrapping.
  • Stop any query or LAT process running on your system.

Tutorial

1

Create the /var/opt/ocient/bootstrap.conf file on the new node using this YAML example. This YAML file should follow a similar format and parameters as the bootstrap.conf files on your other nodes.

Replace the <FIRST_NODE_ADDRESS> with the IP or Hostname of a node running the Administrator Role on your system. If you are not using DNS, use nodeAddress instead of adminHost.

Use a user account with System Administrator privileges for your system for the adminUserName and adminPassword.

YAML

2

Start the new node by using this command.

Shell


This process takes about a minute to complete. This step accepts the node into the system, but it has not been assigned a role or a storage cluster.

3

To validate that the node is on the system, execute this query using the sys.nodes system catalog table.

SQL


Output

SQL


The output shows the new node, named foundation_new in this example, listed as ACCEPTED in the status column.

4

From a SQL command prompt, execute this ALTER CLUSTER SQL statement to add the new node to a storage cluster. In this example, replace storage_cluster_1 and foundation_new with your cluster name and node name, respectively.

SQL


This example adds only one new node. For information on adding multiple nodes, see ALTER CLUSTER ADD PARTICIPANTS.

5

Restart the rolehostd process on all nodes in the system by running the following series of commands at the shell terminal.

First, stop the rolehostd process on all nodes.

Shell


Confirm that the rolehostd process is no longer running.

Shell


Start the rolehostd process on the system again.

Shell

6

You can verify the new node is active by running this SQL query after you connect to the database.

SQL

7

Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see Rebalance System.

Add More Drives to Foundation Nodes

Foundation Nodes can support extra storage by adding additional NVMe drives. This tutorial shows the steps to integrate new drives into an existing Ocient System.

System Considerations

  • For best performance, all Foundation Nodes should have equal storage capacity.
  • No individual cluster should have more than two petabytes of storage.

Prerequisites

  • The process requires systemctl access on your system OS.
  • Any new NVMe drives added to the system must be blank and unpartitioned.

Tutorial

1

Shut down the rolehostd process from the shell prompt.

Shell

2

Install the new storage drives in your system.

3

Restart the node from the shell prompt. 

Shell


The rolehostd process recognizes the new drive.

4

To confirm the new drives are active and running, you can connect to your system and query the sys.storage_device_status system catalog table.

To use this example query, replace <NODE_NAME> with the name of the node where you are adding a drive.

SQL


Output

Text


This output lists the serial numbers of all drives in the foundation0 node, including their statuses.

5

Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see Rebalance System.



Rebalance System

REBALANCE task execution redistributes your data evenly across your segment groups and clusters. Rebalance your system if you have recently installed new hardware, particularly new nodes or drives.

Prerequisites

  • Only one rebalance task can execute at a time on the system. The system logs an error if you try to start a second rebalance task. 
  • You must have the System Administrator role, or be granted the UPDATE privilege on SYSTEM.

Tutorial

1

Execute a SELECT SQL statement to view how the system has distributed the existing data using the sys.tables and sys.nodes system catalog tables.

SQL


Output

Text


The query output shows how data is distributed across your nodes.

2

Execute the REBALANCE task named rebalance_task to reorganize all data in a balanced state using the CREATE TASK SQL statement.

SQL


For more information about the syntax for creating tasks, see Rebuild Tasks and Check Data.

3

View the status of the tasks as the rebalance operation executes using the sys.subtasks system catalog table.

SQL


Output

Text


 The output shows when the REBALANCE task is finished.

4

Execute the query from Step 1 again to see how the system reorganized the data across nodes.

SQL


Output

Text


The output shows the redistributed data.

The REBALANCE task redistributes data across segment groups, so there can be some variance in data across nodes.

Related Links

Replace Nodes