Expand and Rebalance System
If your system reaches capacity, you can add storage by installing additional nodes and drives. This process requires rebalancing the data to distribute the query processing across all available hardware, including newly installed disks or nodes.
This tutorial covers expanding a system with additional storage.
For information on how to replace nodes that have failed or are damaged, see Replace Nodes.
For information on replacing drives that have failed or are damaged, see Replace an NVMe Node Drive.
A core component of an System, the Foundation Nodes store user data in Ocient and perform the bulk of query processing. By adding more Foundation Nodes, a system can support extra storage capacity as well as better performance for query processing.
- No individual cluster should have more than two petabytes of storage.
- Each cluster should have the same number of Foundation Nodes.
- The process requires either the System Administrator role or the granted UPDATE privilege on SYSTEM.
- Any new Foundation Nodes should have the same OS kernel version as other nodes in the system.
- Any new Foundation Nodes should have the same Ocient System version as the other nodes. For more information, see Ocient Application Installation.
- Stop any query or LAT process running on your system.
Create the /var/opt/ocient/bootstrap.conf file on the new node using this YAML example. This YAML file should follow a similar format and parameters as the bootstrap.conf files on your other nodes.
Replace the <FIRST_NODE_ADDRESS> with the IP or Hostname of a node running the Administrator Role on your system. If you are not using DNS, use nodeAddress instead of adminHost.
Use a user account with System Administrator privileges for your system for the adminUserName and adminPassword.
Start the new node by using this command.
This process takes about a minute to complete. This step accepts the node into the system, but it has not been assigned a role or a storage cluster.
To validate that the node is on the system, execute this query using the sys.nodes system catalog table.
Output
The output shows the new node, named foundation_new in this example, listed as ACCEPTED in the status column.
From a SQL command prompt, execute this ALTER CLUSTER SQL statement to add the new node to a storage cluster. In this example, replace storage_cluster_1 and foundation_new with your cluster name and node name, respectively.
This example adds only one new node. For information on adding multiple nodes, see ALTER CLUSTER ADD PARTICIPANTS.
Restart the rolehostd process on all nodes in the system by running the following series of commands at the shell terminal.
First, stop the rolehostd process on all nodes.
Confirm that the rolehostd process is no longer running.
Start the rolehostd process on the system again.
You can verify the new node is active by running this SQL query after you connect to the database.
Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see Rebalance System.
Foundation Nodes can support extra storage by adding additional NVMe drives. This tutorial shows the steps to integrate new drives into an existing Ocient System.
- For best performance, all Foundation Nodes should have equal storage capacity.
- No individual cluster should have more than two petabytes of storage.
- The process requires systemctl access on your system OS.
- Any new NVMe drives added to the system must be blank and unpartitioned.
Shut down the rolehostd process from the shell prompt.
Install the new storage drives in your system.
Restart the node from the shell prompt.
The rolehostd process recognizes the new drive.
To confirm the new drives are active and running, you can connect to your system and query the sys.storage_device_status system catalog table.
To use this example query, replace <NODE_NAME> with the name of the node where you are adding a drive.
Output
This output lists the serial numbers of all drives in the foundation0 node, including their statuses.
Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see Rebalance System.
REBALANCE task execution redistributes your data evenly across your segment groups and clusters. Rebalance your system if you have recently installed new hardware, particularly new nodes or drives.
- Only one rebalance task can execute at a time on the system. The system logs an error if you try to start a second rebalance task.
- You must have the System Administrator role, or be granted the UPDATE privilege on SYSTEM.
Execute a SELECT SQL statement to view how the system has distributed the existing data using the sys.tables and sys.nodes system catalog tables.
Output
The query output shows how data is distributed across your nodes.
Execute the REBALANCE task named rebalance_task to reorganize all data in a balanced state using the CREATE TASK SQL statement.
For more information about the syntax for creating tasks, see Rebuild Tasks and Check Data.
View the status of the tasks as the rebalance operation executes using the sys.subtasks system catalog table.
Output
The output shows when the REBALANCE task is finished.
Execute the query from Step 1 again to see how the system reorganized the data across nodes.
Output
The output shows the redistributed data.
The REBALANCE task redistributes data across segment groups, so there can be some variance in data across nodes.