Expand and Rebalance System

If your system reaches capacity, you can add storage by installing additional nodes and drives. This process requires rebalancing the data to distribute the query processing across all available hardware, including newly installed disks or nodes. 
This tutorial covers expanding a system with additional storage. 
For information on how to replace nodes that have failed or are damaged, see Replace Nodes﻿. 
For information on replacing drives that have failed or are damaged, see Replace an NVMe Node Drive﻿. 
Add Foundation Nodes
A core component of an 
﻿ System, the Foundation Nodes store user data in Ocient and perform the bulk of query processing. By adding more Foundation Nodes, a system can support extra storage capacity as well as better performance for query processing. ﻿
System Considerations
No individual cluster should have more than two petabytes of storage. 
Each cluster should have the same number of Foundation Nodes.
Prerequisites
﻿The process requires either the System Administrator role or the granted UPDATE privilege on SYSTEM. ﻿
Any new Foundation Nodes should have the same OS kernel version as other nodes in the system. ﻿
Any new Foundation Nodes should have the same Ocient System version as the other nodes. For more information, see Ocient Application Installation﻿.
Ensure that your system meets the requirements outlined in Ocient System Bootstrapping﻿.  
Stop any query or LAT process running on your system. 
Tutorial 
1
Create the /var/opt/ocient/bootstrap.conf file on the new node using this YAML example. This YAML file should follow a similar format and parameters as the bootstrap.conf files on your other nodes. 
Replace the <FIRST_NODE_ADDRESS> with the IP or Hostname of a node running the Administrator Role on your system.  If you are not using DNS, use nodeAddress instead of adminHost.
Use a user account with System Administrator privileges for your system for the adminUserName and adminPassword.
YAML
adminHost: <FIRST_NODE_ADDRESS>
adminUserName: my_admin 
adminPassword: example_password
adminHost: <FIRST_NODE_ADDRESS>
adminUserName: my_admin 
adminPassword: example_password
﻿
2
Start the new node by using this command. 
Shell
sudo systemctl start rolehostd
sudo systemctl start rolehostd
﻿
This process takes about a minute to complete. This step accepts the node into the system, but it has not been assigned a role or a storage cluster. 
3
To validate that the node is on the system, execute this query using the sys.nodes system catalog table. 
SQL
SELECT name, status 
FROM sys.nodes 
ORDER BY 1; 
SELECT name, status 
FROM sys.nodes 
ORDER BY 1; 
﻿
Output
SQL
name                               status
--------------------------------------------------------------------
admin01                            ACCEPTED
admin02                            ACCEPTED
admin03                            ACCEPTED
loader01                           ACCEPTED
loader02                           ACCEPTED
loader03                           ACCEPTED
foundation01                       ACCEPTED
foundation02                       ACCEPTED
foundation03                       ACCEPTED
foundation04                       ACCEPTED
foundation05                       ACCEPTED
foundation06                       ACCEPTED
foundation07                       ACCEPTED
foundation08                       ACCEPTED
foundation09                       ACCEPTED
foundation10                       ACCEPTED
foundation11                       ACCEPTED
foundation12                       ACCEPTED
foundation_new                     ACCEPTED
sql01                              ACCEPTED
sql02                              ACCEPTED
name                               status
--------------------------------------------------------------------
admin01                            ACCEPTED
admin02                            ACCEPTED
admin03                            ACCEPTED
loader01                           ACCEPTED
loader02                           ACCEPTED
loader03                           ACCEPTED
foundation01                       ACCEPTED
foundation02                       ACCEPTED
foundation03                       ACCEPTED
foundation04                       ACCEPTED
foundation05                       ACCEPTED
foundation06                       ACCEPTED
foundation07                       ACCEPTED
foundation08                       ACCEPTED
foundation09                       ACCEPTED
foundation10                       ACCEPTED
foundation11                       ACCEPTED
foundation12                       ACCEPTED
foundation_new                     ACCEPTED
sql01                              ACCEPTED
sql02                              ACCEPTED
﻿
The output shows the new node, named foundation_new in this example, listed as ACCEPTED in the status column.
4
From a SQL command prompt, execute this ALTER CLUSTER SQL statement to add the new node to a storage cluster. In this example, replace storage_cluster_1 and foundation_new with your cluster name and node name, respectively. 
SQL
ALTER CLUSTER "storage_cluster_1" 
    ADD PARTICIPANTS "foundation_new";
ALTER CLUSTER "storage_cluster_1" 
    ADD PARTICIPANTS "foundation_new";
﻿
This example adds only one new node. For information on adding multiple nodes, see ALTER CLUSTER ADD PARTICIPANTS﻿. 
5
Restart the rolehostd process on all nodes in the system by running the following series of commands at the shell terminal.
First, stop the rolehostd process on all nodes. 
Shell
sudo systemctl kill -s SIGKILL rolehostd
sudo systemctl kill -s SIGKILL rolehostd
﻿
Confirm that the rolehostd process is no longer running. 
Shell
sudo systemctl status rolehostd
sudo systemctl status rolehostd
﻿
Start the rolehostd process on the system again. 
Shell
sudo systemctl start rolehostd
sudo systemctl start rolehostd
﻿
6
You can verify the new node is active by running this SQL query after you connect to the database. 
SQL
SELECT n.name,
       ns.operational_status
FROM sys.node_status ns
JOIN sys.nodes n ON ns.node_id = n.id
ORDER BY n.name;

name                               operational_status
------------------------------------------------------------------------------------------
admin01                            Active
admin02                            Active
admin03                            Active
loader01                           Active
loader02                           Active
loader03                           Active
foundation01                       Active
foundation02                       Active
foundation03                       Active
foundation04                       Active
foundation05                       Active
foundation06                       Active
foundation07                       Active
foundation08                       Active
foundation09                       Active
foundation10                       Active
foundation11                       Active
foundation12                       Active
foundation_new                     Active
sql01                              Active
sql02                              Active
SELECT n.name,
       ns.operational_status
FROM sys.node_status ns
JOIN sys.nodes n ON ns.node_id = n.id
ORDER BY n.name;

name                               operational_status
------------------------------------------------------------------------------------------
admin01                            Active
admin02                            Active
admin03                            Active
loader01                           Active
loader02                           Active
loader03                           Active
foundation01                       Active
foundation02                       Active
foundation03                       Active
foundation04                       Active
foundation05                       Active
foundation06                       Active
foundation07                       Active
foundation08                       Active
foundation09                       Active
foundation10                       Active
foundation11                       Active
foundation12                       Active
foundation_new                     Active
sql01                              Active
sql02                              Active
﻿
7
Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see Rebalance System﻿.
Add More Drives to Foundation Nodes
Foundation Nodes can support extra storage by adding additional NVMe drives. This tutorial shows the steps to integrate new drives into an existing Ocient System. 
System Considerations
For best performance, all Foundation Nodes should have equal storage capacity. 
No individual cluster should have more than two petabytes of storage. 
Prerequisites
The process requires systemctl access on your system OS. 
Any new NVMe drives added to the system must be blank and unpartitioned. 
Tutorial 
1
Shut down the rolehostd process from the shell prompt. 
Shell
sudo systemctl stop rolehostd
sudo systemctl stop rolehostd
﻿
2
Install the new storage drives in your system. 
3
Restart the node from the shell prompt. ﻿
Shell
sudo systemctl restart rolehostd
sudo systemctl restart rolehostd
﻿
The rolehostd process recognizes the new drive. 
4
To confirm the new drives are active and running, you can connect to your system and query the sys.storage_device_status﻿ system catalog table. 
To use this example query, replace <NODE_NAME> with the name of the node where you are adding a drive.
SQL
SELECT
  n.name AS node_name,
  s.node_id,
  s.id AS serial_number,
  s.pci_address,
  s.device_status,
  s.device_model
FROM
  sys.nodes n
JOIN sys.storage_device_status s
  ON n.id = s.node_id
WHERE
  n.name = '<NODE_NAME>';
SELECT
  n.name AS node_name,
  s.node_id,
  s.id AS serial_number,
  s.pci_address,
  s.device_status,
  s.device_model
FROM
  sys.nodes n
JOIN sys.storage_device_status s
  ON n.id = s.node_id
WHERE
  n.name = '<NODE_NAME>';
﻿
Output
Text
|node_name  |node_id                             |serial_number                       |pci_address                                             |device_status|device_model                           |
|-----------|------------------------------------|------------------------------------|--------------------------------------------------------|-------------|---------------------------------------|
|foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|6156962f-fcf6-4299-bbd7-2618fc6f1d00|/var/opt/ocient/6156962f-fcf6-4299-bbd7-2618fc6f1d00.dat|ACTIVE       |PCIe Data Center SSD INTEL SSDPE2ME800G4|
|foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|e6a4b24f-0d49-4704-b7ce-18af163c0701|/var/opt/ocient/e6a4b24f-0d49-4704-b7ce-18af163c0701.dat|ACTIVE       |PCIe Data Center SSD INTEL SSDPE2ME800G4|

|node_name  |node_id                             |serial_number                       |pci_address                                             |device_status|device_model                           |
|-----------|------------------------------------|------------------------------------|--------------------------------------------------------|-------------|---------------------------------------|
|foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|6156962f-fcf6-4299-bbd7-2618fc6f1d00|/var/opt/ocient/6156962f-fcf6-4299-bbd7-2618fc6f1d00.dat|ACTIVE       |PCIe Data Center SSD INTEL SSDPE2ME800G4|
|foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|e6a4b24f-0d49-4704-b7ce-18af163c0701|/var/opt/ocient/e6a4b24f-0d49-4704-b7ce-18af163c0701.dat|ACTIVE       |PCIe Data Center SSD INTEL SSDPE2ME800G4|
﻿
This output lists the serial numbers of all drives in the foundation0 node, including their statuses. 
5
Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see Rebalance System﻿.
﻿
Rebalance System
REBALANCE task execution redistributes your data evenly across your segment groups and clusters. Rebalance your system if you have recently installed new hardware, particularly new nodes or drives.  
Prerequisites
Only one rebalance task can execute at a time on the system. The system logs an error if you try to start a second rebalance task. 
You must have the System Administrator role, or be granted the UPDATE privilege on SYSTEM.
Tutorial
1
Execute a SELECT SQL statement to view how the system has distributed the existing data using the sys.tables and sys.nodes system catalog tables.
SQL
SELECT t.name AS table_name,
    n.name AS node_name,
    su.used_bytes FROM sys.storage_used su
    JOIN sys.tables t ON t.id = su.table_id
    JOIN sys.nodes n ON n.id = su.node_id
ORDER BY t.name,
    n.name;
SELECT t.name AS table_name,
    n.name AS node_name,
    su.used_bytes FROM sys.storage_used su
    JOIN sys.tables t ON t.id = su.table_id
    JOIN sys.nodes n ON n.id = su.node_id
ORDER BY t.name,
    n.name;
﻿
Output
Text
+-------------------------+-----------+--------------+
| table_name              | node_name |   used_bytes |
|-------------------------+-----------+--------------|
| table0                  | lts0      |     48000000 |
| table0                  | lts1      |     11000000 |
| table0                  | lts2      |     79000000 |
| table0                  | lts3      |     73000000 |
| table0                  | lts4      |     85000000 |
| table0                  | lts5      |     50000000 |
| table0                  | lts6      |     51000000 |
| table0                  | lts7      |     41000000 |
| table1                  | lts0      |     66000000 |
| table1                  | lts1      |     77000000 |
| table1                  | lts2      |     73000000 |
| table1                  | lts3      |     43000000 |
| table1                  | lts4      |     76000000 |
| table1                  | lts5      |     51000000 |
| table1                  | lts6      |     71000000 |
| table1                  | lts7      |    105000000 |
+-------------------------+-----------+--------------+
| table_name              | node_name |   used_bytes |
|-------------------------+-----------+--------------|
| table0                  | lts0      |     48000000 |
| table0                  | lts1      |     11000000 |
| table0                  | lts2      |     79000000 |
| table0                  | lts3      |     73000000 |
| table0                  | lts4      |     85000000 |
| table0                  | lts5      |     50000000 |
| table0                  | lts6      |     51000000 |
| table0                  | lts7      |     41000000 |
| table1                  | lts0      |     66000000 |
| table1                  | lts1      |     77000000 |
| table1                  | lts2      |     73000000 |
| table1                  | lts3      |     43000000 |
| table1                  | lts4      |     76000000 |
| table1                  | lts5      |     51000000 |
| table1                  | lts6      |     71000000 |
| table1                  | lts7      |    105000000 |
﻿
The query output shows how data is distributed across your nodes. 
2
Execute the REBALANCE task named rebalance_task to reorganize all data in a balanced state using the CREATE TASK SQL statement.
SQL
CREATE TASK rebalance_task TYPE REBALANCE;
CREATE TASK rebalance_task TYPE REBALANCE;
﻿
For more information about the syntax for creating tasks, see Rebuild Tasks and Check Data﻿. 
3
View the status of the tasks as the rebalance operation executes using the sys.subtasks system catalog table.
SQL
SELECT id,
    name,
    start_time,
    task_type,
    execution_type,
    status,
    details
FROM sys.subtasks
WHERE task_type = 'rebalance'
ORDER BY start_time DESC;
SELECT id,
    name,
    start_time,
    task_type,
    execution_type,
    status,
    details
FROM sys.subtasks
WHERE task_type = 'rebalance'
ORDER BY start_time DESC;
﻿
Output
Text
+--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+
| id                                   | name   | start_time                 | task_type   | execution_type          | status   | details                                          |
|--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------|
| ed68a070-d7f7-4c50-9b94-84b04b4503c2 |        | 2024-04-17 21:44:20.268000 | rebalance   | intra_cluster_rebalance | COMPLETE |                                                  |
| 9e6c3374-6e9c-4a84-9bf8-ba1db3d866eb |        | 2024-04-17 21:44:20.268000 | rebalance   | intra_cluster_rebalance | COMPLETE |                                                  |
| aebb75d8-e066-4813-b7e8-c272e5a3b8e9 |        | 2024-04-17 21:44:14.768000 | rebalance   | inter_cluster_rebalance | COMPLETE |                                                  |
| 5cec5095-8b3a-4868-a10d-35fe6a32e998 |        | 2024-04-17 21:44:14.564000 | rebalance   | NULL                    | COMPLETE | Task finalizing due to terminal status: COMPLETE |
+--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+
+--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+
| id                                   | name   | start_time                 | task_type   | execution_type          | status   | details                                          |
|--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------|
| ed68a070-d7f7-4c50-9b94-84b04b4503c2 |        | 2024-04-17 21:44:20.268000 | rebalance   | intra_cluster_rebalance | COMPLETE |                                                  |
| 9e6c3374-6e9c-4a84-9bf8-ba1db3d866eb |        | 2024-04-17 21:44:20.268000 | rebalance   | intra_cluster_rebalance | COMPLETE |                                                  |
| aebb75d8-e066-4813-b7e8-c272e5a3b8e9 |        | 2024-04-17 21:44:14.768000 | rebalance   | inter_cluster_rebalance | COMPLETE |                                                  |
| 5cec5095-8b3a-4868-a10d-35fe6a32e998 |        | 2024-04-17 21:44:14.564000 | rebalance   | NULL                    | COMPLETE | Task finalizing due to terminal status: COMPLETE |
+--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+
﻿
 The output shows when the REBALANCE task is finished. 
4
Execute the query from Step 1 again to see how the system reorganized the data across nodes. 
SQL
SELECT t.name,
    n.name,
    su.used_bytes 
FROM sys.storage_used su
    JOIN sys.tables t ON t.id = su.table_id
    JOIN sys.nodes n ON n.id = su.node_id
ORDER BY t.name,
    n.name;
SELECT t.name,
    n.name,
    su.used_bytes 
FROM sys.storage_used su
    JOIN sys.tables t ON t.id = su.table_id
    JOIN sys.nodes n ON n.id = su.node_id
ORDER BY t.name,
    n.name;
﻿
Output
Text
+-------------------------+----------+--------------+
| name                    | name_1   |   used_bytes |
|-------------------------+----------+--------------|
| table0                  | lts0     |     48000000 |
| table0                  | lts1     |     39000000 |
| table0                  | lts2     |     71000000 |
| table0                  | lts3     |     53000000 |
| table0                  | lts4     |     85000000 |
| table0                  | lts5     |     50000000 |
| table0                  | lts6     |     51000000 |
| table0                  | lts7     |     41000000 |
| table1                  | lts0     |     66000000 |
| table1                  | lts1     |     77000000 |
| table1                  | lts2     |     73000000 |
| table1                  | lts3     |     43000000 |
| table1                  | lts4     |     76000000 |
| table1                  | lts5     |     68000000 |
| table1                  | lts6     |     71000000 |
| table1                  | lts7     |     88000000 |
+-------------------------+----------+--------------+
| name                    | name_1   |   used_bytes |
|-------------------------+----------+--------------|
| table0                  | lts0     |     48000000 |
| table0                  | lts1     |     39000000 |
| table0                  | lts2     |     71000000 |
| table0                  | lts3     |     53000000 |
| table0                  | lts4     |     85000000 |
| table0                  | lts5     |     50000000 |
| table0                  | lts6     |     51000000 |
| table0                  | lts7     |     41000000 |
| table1                  | lts0     |     66000000 |
| table1                  | lts1     |     77000000 |
| table1                  | lts2     |     73000000 |
| table1                  | lts3     |     43000000 |
| table1                  | lts4     |     76000000 |
| table1                  | lts5     |     68000000 |
| table1                  | lts6     |     71000000 |
| table1                  | lts7     |     88000000 |
﻿
The output shows the redistributed data. 
The REBALANCE task redistributes data across segment groups, so there can be some variance in data across nodes. 
Related Links
﻿Replace Nodes﻿﻿
﻿Replace an NVMe Node Drive﻿﻿
Updated 14 May 2024
Did this page help you?
Replace an NVMe Node Drive
Replace Nodes