> ## Documentation Index > Fetch the complete documentation index at: https://docs.ocient.com/llms.txt > Use this file to discover all available pages before exploring further. # Guide to Rebuilding Segments export const Ocient = "Ocient®"; Data in an {Ocient} System is erasure coded to provide resilience to disk and Foundation Node failures. The resilience of a system depends on the parity width of the storage space, which represents the number of associated disks (or nodes) that can fail without interruption of service or data loss. For details about storage spaces and parity width, see [Configure Storage Spaces](/configure-storage-spaces). ## Data Segment Statuses You can check the status of your system segment groups by querying the `sys.segment_groups` system catalog table. For details, see [System Catalog](/system-catalog). This table describes the states for data segments. | **Status** | **Description** | **Recovery Process** | | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | INTACT | The normal and operable status. | No recovery needed. | | DAMAGED | The segment failed a checksum, meaning the segment data is corrupted and unusable. | If you have sufficient parity width, you can recover a damaged segment by invoking a rebuild segment task. | | MISSING | The segment is on a node or disk that is currently offline. If the node or disk rejoins the cluster, it can transition to the INTACT status. | When a disk or Foundation Node is permanently removed, you can perform a rebuild task to recover the data. This requires sufficient parity width. | | REBUILDING | The segment is in recovery after damage or missing segment data. | Recovery is already in progress. | ## Recovery Considerations If a segment has the DAMAGED or MISSING status, queries can proceed by reconstructing the data on demand using the remaining erasure-coded data in the segment group. Having a non-INTACT status means that input or output (I/O) performance is significantly reduced. To restore full performance, you need to run a segment rebuild task. Segment rebuilding is not automatic. The system administrator must manually invoke it. A segment rebuild fails, and data is lost completely if the number of segments with the DAMAGED status in a segment group exceeds the parity width of the storage space. To avoid data loss: * Provision the parity width of the storage space at or above the number of expected concurrent node failures. * Rebuild damaged or permanently missing segments as soon as possible. ## Checking for Abnormal Segments You can find any segments that need a rebuild by querying for segment groups with an abnormal status. **Examples** **Finding Faulty Segment Groups** This example query finds any segment groups with the DAMAGED or MISSING status. ```sql SQL theme={null} SELECT * FROM sys.segment_groups WHERE status IN ('DAMAGED', 'MISSING'); ``` *Output* ```none Text theme={null} | "id" | "cluster_id" | "segment_type" | "status" | "primary_owner" | "loader_id" | "table_id" | "scope_id" | "block_size" | "begin_time" | "end_time" | "coding_algorithm" | "coding_block_size" | "coding_threshold" | "coding_width" | "replication" | "parity_cycle" | "created_time" | "rolehostd_version" | "commit_hash" | "timestamp" | "build_user" | "depth" | "removal_type" | |----------------------|----------------------------------------|----------------|----------|-----------------|----------------------------------------|----------------------------------------|----------------------------------------|--------------|--------------|------------|--------------------|---------------------|--------------------|----------------|---------------|----------------|-------------------------|---------------------|--------------------------------------------|-------------------|--------------|---------|----------------| | "720576045199095878" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT | DAMAGED | | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | bd18d33b-aae9-4be2-a2f4-76ecdc34c2ba | "8bc119d2-875a-47e4-b016-87fde83e77d6" | 4096 | 0 | 1 | XOR_PARITY | 4096 | 2 | 3 | 1 | 1 | 2025-02-26 20:23:34.274 | "25.0.0" | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" | | 0 | NOT_REMOVED | | "716072445571725367" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT | DAMAGED | | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | "9fe79009-ee75-4429-b29b-3a614a166751" | a8c805d0-2144-46a7-8374-52cb88d64244 | 4096 | 0 | 1 | XOR_PARITY | 4096 | 2 | 3 | 1 | 1 | 2025-02-26 20:23:19.447 | "25.0.0" | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" | | 0 | NOT_REMOVED | ``` **Finding Clusters and Nodes with Faulty Segment Groups** Inspect the count of damaged groups by cluster and node. ```sql SQL theme={null} SELECT c.name AS cluster_name, n.name AS node_name, g.status AS segment_group_status, seg.status AS segment_status, seg.kind, COUNT(*) AS segment_count FROM sys.segment_groups g LEFT JOIN sys.clusters c ON c.id = g.cluster_id LEFT JOIN sys.stored_segments seg ON seg.segment_group_id = g.id LEFT JOIN sys.nodes n ON n.id = seg.node_id WHERE g.status <> 'INTACT' AND (seg.status <> 'INTACT' OR seg.status IS NULL) GROUP BY 1,2,3,4,5 ORDER BY 1,2,3,4,5; ``` *Output* ```none Text theme={null} | "cluster_name" | "node_name" | "segment_group_status" | "segment_status" | "kind" | "segment_count" | |--------------------|-------------|------------------------|------------------|---------|-----------------| | foundation_cluster | foundation0 | DAMAGED | | VIRTUAL | 2 | ``` ## Starting a Segment Rebuild Task A user with System Administrator privileges can start a segment rebuild using the `CREATE TASK TYPE REBUILD` SQL statement. You cannot cancel a segment rebuild task after it is started. Most commonly, a rebuild task repairs all damaged or missing segments across the system. **Example** Create a rebuild task. ```sql SQL theme={null} CREATE TASK TYPE REBUILD; ``` The system can continue to perform queries while rebuilding segments, but the process can impact I/O performance. ### Advanced Rebuild Commands Rebuild tasks can also execute on specific Foundation Nodes or clusters. For information on fine-tuning rebuild tasks, see [CREATE TASK](/distributed-tasks#create-task). ## Checking Rebuild Task Status Monitor the status of current and past segment rebuild tasks from the `sys.subtasks` system catalog table. For details, see [System Catalog](/system-catalog). ```sql SQL theme={null} SELECT * FROM sys.subtasks WHERE task_type = 'rebuild'; ``` This table describes the statuses for a rebuild task. | **Status** | **Status Detail** | **Description** | **Next Steps** | | ---------- | ---------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | complete | no\_work | The segment groups were already available and healthy. No rebuilding was needed. | None | | complete | complete | Rebuild completed successfully. | None | | running | rebuild\_in\_progress | Rebuild is in process. | You can monitor the progress of the rebuild task by checking the JSON dictionary in the `details` column of the `sys.subtasks` system catalog table. | | failed | rebuild\_not\_possible | The number of missing or damaged segments exceeds the parity width, and the data cannot be recovered currently. | If missing segments are available on an offline drive, you can attempt another rebuild task when that drive is made available to the system.
Otherwise, you cannot recover the data. | | failed | rebuild\_no\_space | There is not enough space available to rebuild the segment. | You can complete the rebuild by truncating other data to free up space. | | failed | failed\_on\_node | The cluster lost its connection to the node before the rebuild was completed. | This is a transient error. You can retry the rebuild task. | | failed | error | An unexpected internal error occurred. | Review error message details using the `rolehostd` logs. For details, see [Log Monitoring](/log-monitoring). | ## Related Links [CREATE TASK](/distributed-tasks) [Erasure Coding](/compute-adjacent-input-and-output-on-large-working-sets#erasure-coding)