> ## Documentation Index > Fetch the complete documentation index at: https://docs.ocient.com/llms.txt > Use this file to discover all available pages before exploring further. # Expand and Rebalance System export const Ocient = "Ocient®"; If your system needs more resources, you can expand it by installing additional nodes and drives. The steps for expanding a system depend on the type of node you are adding. These sections explain the steps for adding each {Ocient} node type. Foundation Nodes include additional steps for adding more storage drives and rebalancing the system. For information on how to replace nodes that have failed or are damaged, see [Replace Nodes](/replace-nodes). For information on replacing drives that have failed or are damaged, see [Replace an NVMe Node Drive](/replace-an-nvme-node-drive). ## Prerequisites These prerequisites apply to adding any node type: * The process requires either the System Administrator role or the granted UPDATE privilege on SYSTEM. * Any new nodes should have the same OS kernel version as other nodes in the system. * Any new nodes should have the same Ocient System version as the other nodes. For details, see [Ocient Application Installation](/ocient-application-installation). * Ensure that your system meets the requirements outlined in [Ocient System Bootstrapping](/ocient-system-bootstrapping). * Before adding nodes, stop any query or loading process running on your system. ## Add Foundation Nodes A core component of an Ocient System, the Foundation Nodes store user data in Ocient and perform the bulk of query processing. By adding more Foundation Nodes, a system can support extra storage capacity as well as better performance for query processing. This process requires rebalancing the data to distribute the query processing across all available hardware, including newly installed disks or nodes.  ### System Considerations * No individual cluster should have more than two petabytes of storage. * Each cluster should have the same number of Foundation Nodes. ### Tutorial Create the `/var/opt/ocient/bootstrap.conf` file on the new node using this YAML example. This YAML file should follow a similar format and parameters as the `bootstrap.conf` files on your other nodes. Replace the `` with the IP or Hostname of a node running the Administrator Role on your system. If you are not using DNS, use `nodeAddress` instead of `adminHost`. Use a user account with System Administrator privileges for your system for the `adminUserName` and `adminPassword`. ```yaml YAML theme={null} adminHost: adminUserName: my_admin adminPassword: example_password ``` Start the new node by using this command. ```shell Shell theme={null} sudo systemctl start rolehostd ``` This process takes about a minute to complete. This step accepts the node into the system, but it has not been assigned a role or a storage cluster. To validate that the node is on the system, execute this query using the `sys.nodes` system catalog table. ```sql SQL theme={null} SELECT name, status FROM sys.nodes ORDER BY 1; ``` *Output* ```sql SQL theme={null} name status -------------------------------------------------------------------- admin01 ACCEPTED admin02 ACCEPTED admin03 ACCEPTED loader01 ACCEPTED loader02 ACCEPTED loader03 ACCEPTED foundation01 ACCEPTED foundation02 ACCEPTED foundation03 ACCEPTED foundation04 ACCEPTED foundation05 ACCEPTED foundation06 ACCEPTED foundation07 ACCEPTED foundation08 ACCEPTED foundation09 ACCEPTED foundation10 ACCEPTED foundation11 ACCEPTED foundation12 ACCEPTED foundation_new ACCEPTED sql01 ACCEPTED sql02 ACCEPTED ``` The output shows the new node, named `foundation_new` in this example, listed as `ACCEPTED` in the `status` column. Execute this `ALTER CLUSTER` SQL statement to add the new node to a storage cluster. In this example, replace `storage_cluster_1` and `foundation_new` with your cluster name and node name, respectively. ```sql SQL theme={null} ALTER CLUSTER "storage_cluster_1" ADD PARTICIPANTS "foundation_new"; ``` This example adds only one new node. For information on adding multiple nodes, see [ALTER CLUSTER ADD PARTICIPANTS](/cluster-and-node-management#alter-cluster). Restart the `rolehostd` process on all nodes in the system by running the following series of commands at the shell terminal. First, stop the `rolehostd` process on all nodes. ```shell Shell theme={null} sudo systemctl kill -s SIGKILL rolehostd ``` Confirm that the `rolehostd` process is no longer running. ```shell Shell theme={null} sudo systemctl status rolehostd ``` Start the `rolehostd` process on the system again. ```shell Shell theme={null} sudo systemctl start rolehostd ``` You can verify the new node is active by executing this SQL query after you connect to the database. ```sql SQL theme={null} SELECT n.name, ns.operational_status FROM sys.node_status ns JOIN sys.nodes n ON ns.node_id = n.id ORDER BY n.name; name operational_status ------------------------------------------------------------------------------------------ admin01 Active admin02 Active admin03 Active loader01 Active loader02 Active loader03 Active foundation01 Active foundation02 Active foundation03 Active foundation04 Active foundation05 Active foundation06 Active foundation07 Active foundation08 Active foundation09 Active foundation10 Active foundation11 Active foundation12 Active foundation_new Active sql01 Active sql02 Active ``` Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see [Rebalance System](#rebalance-system). ## Add More Drives to Foundation Nodes Foundation Nodes can support extra storage by adding additional NVMe drives. This tutorial shows the steps to integrate new drives into an existing Ocient System. ### System Considerations * For best performance, all Foundation Nodes should have equal storage capacity. * No individual cluster should have more than two petabytes of storage. ### Prerequisites * The process requires `systemctl` access on your system OS. * Any new NVMe drives added to the system must be blank and unpartitioned. ### Tutorial Shut down the `rolehostd` process from the shell prompt. ```shell Shell theme={null} sudo systemctl stop rolehostd ``` Install the new storage drives in your system. Restart the node from the shell prompt. ```shell Shell theme={null} sudo systemctl restart rolehostd ``` The `rolehostd` process recognizes the new drive. To confirm the new drives are active and running, you can connect to your system and query the [sys.storage\_device\_status](/system-catalog#sys-storage_device_status) system catalog table. To use this example query, replace `` with the name of the node where you are adding a drive. ```sql SQL theme={null} SELECT n.name AS node_name, s.node_id, s.id AS serial_number, s.pci_address, s.device_status, s.device_model FROM sys.nodes n JOIN sys.storage_device_status s ON n.id = s.node_id WHERE n.name = ''; ``` *Output* ```none Text theme={null} |node_name |node_id |serial_number |pci_address |device_status|device_model | |-----------|------------------------------------|------------------------------------|--------------------------------------------------------|-------------|---------------------------------------| |foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|6156962f-fcf6-4299-bbd7-2618fc6f1d00|/var/opt/ocient/6156962f-fcf6-4299-bbd7-2618fc6f1d00.dat|ACTIVE |PCIe Data Center SSD INTEL SSDPE2ME800G4| |foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|e6a4b24f-0d49-4704-b7ce-18af163c0701|/var/opt/ocient/e6a4b24f-0d49-4704-b7ce-18af163c0701.dat|ACTIVE |PCIe Data Center SSD INTEL SSDPE2ME800G4| ``` This output lists the serial numbers of all drives in the `foundation0` node, including their statuses. Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see [Rebalance System](#rebalance-system). ## Rebalance System `REBALANCE` task execution redistributes your data evenly across your segment groups and clusters. Rebalance your system if you have recently installed new hardware, particularly new nodes or drives. ### System Considerations Currently, the `REBALANCE` task does not move damaged segment groups. If you suspect that you might have a significant number of damaged segment groups, you can execute these steps to check and fix the groups: 1. Ensure all nodes are online. 2. Check whether you have damaged segment groups using the `sys.segment_groups` system catalog table. If damaged segment groups are present, execute a `REBUILD` task to rebuild the groups before rebalancing the system. For details, see [Guide to Rebuilding Segments](/guide-to-rebuilding-segments). 3. Execute the `REBALANCE` task using this tutorial. You can find information in the `sys.degraded_segment_groups` system catalog table to identify the damaged segment groups that need fixing. This table shows all segment groups with the `DAMAGED` or `UNAVAILABLE` state. You can also check the `sys.storage_used` system catalog table, which shows approximately the same number of used bytes `used_bytes` for each node entry with the same table after the rebalance execution. ### Prerequisites * Only one rebalance task can execute at a time on the system. The system logs an error if you try to start a second rebalance task.  * You must have the System Administrator role, or be granted the UPDATE privilege on SYSTEM. ### Tutorial Execute a `SELECT` SQL statement to view how the system has distributed the existing data using the `sys.tables` and `sys.nodes` system catalog tables. ```sql SQL theme={null} SELECT t.name AS table_name, n.name AS node_name, su.used_bytes FROM sys.storage_used su JOIN sys.tables t ON t.id = su.table_id JOIN sys.nodes n ON n.id = su.node_id ORDER BY t.name, n.name; ``` *Output* ```none Text theme={null} +-------------------------+-----------+--------------+ | table_name | node_name | used_bytes | |-------------------------+-----------+--------------| | table0 | lts0 | 48000000 | | table0 | lts1 | 11000000 | | table0 | lts2 | 79000000 | | table0 | lts3 | 73000000 | | table0 | lts4 | 85000000 | | table0 | lts5 | 50000000 | | table0 | lts6 | 51000000 | | table0 | lts7 | 41000000 | | table1 | lts0 | 66000000 | | table1 | lts1 | 77000000 | | table1 | lts2 | 73000000 | | table1 | lts3 | 43000000 | | table1 | lts4 | 76000000 | | table1 | lts5 | 51000000 | | table1 | lts6 | 71000000 | | table1 | lts7 | 105000000 | ``` The query output shows how data is distributed across your nodes. Execute the `REBALANCE` task named `rebalance_task` to reorganize all data in a balanced state using the `CREATE TASK` SQL statement. ```sql SQL theme={null} CREATE TASK rebalance_task TYPE REBALANCE; ``` For details about the syntax for creating tasks, see [Distributed Tasks](/distributed-tasks). View the status of the tasks as the rebalance operation executes using the `sys.subtasks` system catalog table. ```sql SQL theme={null} SELECT id, name, start_time, task_type, execution_type, status, details FROM sys.subtasks WHERE task_type = 'rebalance' ORDER BY start_time DESC; ``` *Output* ```none Text theme={null} +--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+ | id | name | start_time | task_type | execution_type | status | details | |--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------| | ed68a070-d7f7-4c50-9b94-84b04b4503c2 | | 2024-04-17 21:44:20.268000 | rebalance | intra_cluster_rebalance | COMPLETE | | | 9e6c3374-6e9c-4a84-9bf8-ba1db3d866eb | | 2024-04-17 21:44:20.268000 | rebalance | intra_cluster_rebalance | COMPLETE | | | aebb75d8-e066-4813-b7e8-c272e5a3b8e9 | | 2024-04-17 21:44:14.768000 | rebalance | inter_cluster_rebalance | COMPLETE | | | 5cec5095-8b3a-4868-a10d-35fe6a32e998 | | 2024-04-17 21:44:14.564000 | rebalance | NULL | COMPLETE | Task finalizing due to terminal status: COMPLETE | +--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+ ```  The output shows when the `REBALANCE` task is finished. When the `REBALANCE` task runs, segments are in the `REBUILDING` state. After the Ocient System completes this task, all segments should transition to the `INTACT` state. Execute the query from Step 1 again to see how the system reorganized the data across nodes. ```sql SQL theme={null} SELECT t.name, n.name, su.used_bytes FROM sys.storage_used su JOIN sys.tables t ON t.id = su.table_id JOIN sys.nodes n ON n.id = su.node_id ORDER BY t.name, n.name; ``` *Output* ```none Text theme={null} +-------------------------+----------+--------------+ | name | name_1 | used_bytes | |-------------------------+----------+--------------| | table0 | lts0 | 48000000 | | table0 | lts1 | 39000000 | | table0 | lts2 | 71000000 | | table0 | lts3 | 53000000 | | table0 | lts4 | 85000000 | | table0 | lts5 | 50000000 | | table0 | lts6 | 51000000 | | table0 | lts7 | 41000000 | | table1 | lts0 | 66000000 | | table1 | lts1 | 77000000 | | table1 | lts2 | 73000000 | | table1 | lts3 | 43000000 | | table1 | lts4 | 76000000 | | table1 | lts5 | 68000000 | | table1 | lts6 | 71000000 | | table1 | lts7 | 88000000 | ``` The output shows the redistributed data. The `REBALANCE` task redistributes data across segment groups, so there can be some variance in data across nodes. ## Add Loader Nodes Adding more Loader Nodes to your system can improve the throughput of data loading and resiliency against loading failure. By having extra Loader Nodes, you can also dedicate sets of nodes for specific pipelines. ### System Considerations Before starting this process, ensure you meet the requirements in the [Prerequisites](#prerequisites) section. ### Tutorial Stop any active data pipelines. Execute this SQL statement by replacing `pipeline_name` with the name of your data pipeline. ```sql SQL theme={null} STOP PIPELINE ; ``` Create the `/var/opt/ocient/bootstrap.conf` file on the new Loader Node using this YAML example. This YAML file should follow a similar format and parameters to the `bootstrap.conf` file on your other nodes. Replace the `` with the IP address or hostname of a node running the Administrator Role on your system. If you are not using a DNS, use `nodeAddress` instead of `adminHost`. Specify a user account with System Administrator privileges for your system for the username `adminUserName` and password `adminPassword`. ```yaml YAML theme={null} adminHost: adminUserName: my_admin adminPassword: example_password ``` Start the new node by using this command. ```shell Shell theme={null} sudo systemctl start rolehostd ``` This process takes about a minute to complete. This step accepts the node into the system, but does not assign a role. Add the `streamloader` role to the new node. Execute this SQL statement by replacing `` with the new node name. ```sql SQL theme={null} ALTER NODE ADD ROLE streamloader; ``` Restart the `rolehostd` process on the replacement node by running this command at the shell terminal on the replacement node. ```shell Shell theme={null} sudo systemctl restart rolehostd ``` Restart the pipeline. Execute this SQL statement by replacing `pipeline_name` with the name of your data pipeline. Optionally, specify the Loader Node by name with the `USING LOADERS` keywords to prioritize it for this pipeline. ```shell Shell theme={null} START PIPELINE USING LOADERS ; ``` If you use the legacy LAT service, you must stop loading and copy the LAT configuration files to any new Loader Nodes. For details, see [Configure the LAT Service](/replace-nodes#step-4-configure-the-lat-service). This step is unnecessary for systems that use Ocient data pipelines for loading. ## Add SQL Nodes Adding more SQL Nodes to your system can improve query optimization and processing, particularly for aggregation and join operations. Extra SQL Nodes also provide system resiliency, especially when assigned the `admin` role. ### System Considerations Before starting this process, ensure you meet the requirements in the [Prerequisites](#prerequisites) section. ### Tutorial Create the `/var/opt/ocient/bootstrap.conf` file on the new SQL Node using this YAML example. This YAML file should follow a similar format and parameters to the `bootstrap.conf` file on your other nodes. Replace the `` with the IP address or hostname of a node running the Administrator Role on your system. If you are not using a DNS, use `nodeAddress` instead of `adminHost`. Specify a user account with System Administrator privileges for your system for the username `adminUserName` and password `adminPassword`. ```yaml YAML theme={null} adminHost: adminUserName: my_admin adminPassword: example_password ``` Start the new node by using this command. ```shell Shell theme={null} sudo systemctl start rolehostd ``` This process takes about a minute to complete. This step accepts the node into the system, but does not assign a role. Add the `sql` role to the new node. Execute this SQL statement by replacing ``with the new node name. ```sql SQL theme={null} ALTER NODE ADD ROLE sql; ``` Optionally, you can also assign the `admin` role to the SQL Node. At least one SQL Node must always fulfill this role. By default, the system assigns the `admin` role to the first SQL Node in the system. For details about the `admin` role, see [Node Configuration with the Administrator Role](/node-configuration-with-the-administrator-role). Execute this SQL statement by replacing `` with the name of your new node. ```sql SQL theme={null} ALTER NODE ADD ROLE admin; ``` Restart the `rolehostd` process on the replacement node by running this command at the shell terminal on the replacement node. ```shell Shell theme={null} sudo systemctl restart rolehostd ``` Assign the new SQL Node to a connectivity pool with the [ALTER CONNECTIVITY\_POOL](/cluster-and-node-management#alter-connectivity_pool-add-participants) statement. In this example, the statement assigns the SQL Node `sql2` to the connectivity pool `cp1` with the IP address `111.1.1.1` and port number `4050` for listening. Specify the local IP address and port number `4050` to return to the client. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL cp1 ADD PARTICIPANTS( NODE sql2 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050); ``` ## Related Links [Replace Nodes](/replace-nodes) [Replace an NVMe Node Drive](/replace-an-nvme-node-drive) [System Catalog](/system-catalog)