Guide to Rebuilding Segments - Ocient Documentation

Data in an System is erasure coded to provide resilience to disk and Foundation Node failures. The resilience of a system depends on the parity width of the storage space, which represents the number of associated disks (or nodes) that can fail without interruption of service or data loss. For details about storage spaces and parity width, see Configure Storage Spaces.

Data Segment Statuses

You can check the status of your system segment groups by querying the sys.segment_groups system catalog table. For details, see System Catalog. This table describes the states for data segments.

Status	Description	Recovery Process
INTACT	The normal and operable status.	No recovery needed.
DAMAGED	The segment failed a checksum, meaning the segment data is corrupted and unusable.	If you have sufficient parity width, you can recover a damaged segment by invoking a rebuild segment task.
MISSING	The segment is on a node or disk that is currently offline. If the node or disk rejoins the cluster, it can transition to the INTACT status.	When a disk or Foundation Node is permanently removed, you can perform a rebuild task to recover the data. This requires sufficient parity width.
REBUILDING	The segment is in recovery after damage or missing segment data.	Recovery is already in progress.

Recovery Considerations

If a segment has the DAMAGED or MISSING status, queries can proceed by reconstructing the data on demand using the remaining erasure-coded data in the segment group. Having a non-INTACT status means that input or output (I/O) performance is significantly reduced. To restore full performance, you need to run a segment rebuild task. Segment rebuilding is not automatic. The system administrator must manually invoke it. A segment rebuild fails, and data is lost completely if the number of segments with the DAMAGED status in a segment group exceeds the parity width of the storage space. To avoid data loss:

Provision the parity width of the storage space at or above the number of expected concurrent node failures.
Rebuild damaged or permanently missing segments as soon as possible.

Checking for Abnormal Segments

You can find any segments that need a rebuild by querying for segment groups with an abnormal status. Examples Finding Faulty Segment Groups This example query finds any segment groups with the DAMAGED or MISSING status.

SQL

SELECT * FROM sys.segment_groups WHERE status IN ('DAMAGED', 'MISSING');

Output

Text

|         "id"         |              "cluster_id"              | "segment_type" | "status" | "primary_owner" |              "loader_id"               |               "table_id"               |               "scope_id"               | "block_size" | "begin_time" | "end_time" | "coding_algorithm" | "coding_block_size" | "coding_threshold" | "coding_width" | "replication" | "parity_cycle" |     "created_time"      | "rolehostd_version" |               "commit_hash"                |    "timestamp"    | "build_user" | "depth" | "removal_type" |
|----------------------|----------------------------------------|----------------|----------|-----------------|----------------------------------------|----------------------------------------|----------------------------------------|--------------|--------------|------------|--------------------|---------------------|--------------------|----------------|---------------|----------------|-------------------------|---------------------|--------------------------------------------|-------------------|--------------|---------|----------------|
| "720576045199095878" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT    | DAMAGED  |                 | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | bd18d33b-aae9-4be2-a2f4-76ecdc34c2ba   | "8bc119d2-875a-47e4-b016-87fde83e77d6" |         4096 |            0 |          1 | XOR_PARITY         |                4096 |                  2 |              3 |             1 |              1 | 2025-02-26 20:23:34.274 | "25.0.0"            | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" |              |       0 | NOT_REMOVED    |
| "716072445571725367" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT    | DAMAGED  |                 | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | "9fe79009-ee75-4429-b29b-3a614a166751" | a8c805d0-2144-46a7-8374-52cb88d64244   |         4096 |            0 |          1 | XOR_PARITY         |                4096 |                  2 |              3 |             1 |              1 | 2025-02-26 20:23:19.447 | "25.0.0"            | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" |              |       0 | NOT_REMOVED    |

Finding Clusters and Nodes with Faulty Segment Groups Inspect the count of damaged groups by cluster and node.

SQL

SELECT
  c.name AS cluster_name,
  n.name AS node_name,
  g.status AS segment_group_status,
  seg.status AS segment_status,
  seg.kind,
  COUNT(*) AS segment_count
FROM
  sys.segment_groups g
LEFT JOIN
  sys.clusters c
  ON c.id = g.cluster_id
LEFT JOIN
  sys.stored_segments seg
  ON seg.segment_group_id = g.id
LEFT JOIN
  sys.nodes n
  ON n.id = seg.node_id
WHERE
  g.status <> 'INTACT'
  AND (seg.status <> 'INTACT' OR seg.status IS NULL)
  GROUP BY 1,2,3,4,5
  ORDER BY 1,2,3,4,5;

Output

Text

|   "cluster_name"   | "node_name" | "segment_group_status" | "segment_status" | "kind"  | "segment_count" |
|--------------------|-------------|------------------------|------------------|---------|-----------------|
| foundation_cluster | foundation0 | DAMAGED                |                  | VIRTUAL |               2 |

Starting a Segment Rebuild Task

A user with System Administrator privileges can start a segment rebuild using the CREATE TASK TYPE REBUILD SQL statement. You cannot cancel a segment rebuild task after it is started. Most commonly, a rebuild task repairs all damaged or missing segments across the system. Example Create a rebuild task.

SQL

CREATE TASK TYPE REBUILD;

The system can continue to perform queries while rebuilding segments, but the process can impact I/O performance.

Advanced Rebuild Commands

Rebuild tasks can also execute on specific Foundation Nodes or clusters. For information on fine-tuning rebuild tasks, see CREATE TASK.

Checking Rebuild Task Status

Monitor the status of current and past segment rebuild tasks from the sys.subtasks system catalog table. For details, see System Catalog.

SQL

SELECT * FROM sys.subtasks WHERE task_type = 'rebuild';

This table describes the statuses for a rebuild task.

Status	Status Detail	Description	Next Steps
complete	no_work	The segment groups were already available and healthy. No rebuilding was needed.	None
complete	complete	Rebuild completed successfully.	None
running	rebuild_in_progress	Rebuild is in process.	You can monitor the progress of the rebuild task by checking the JSON dictionary in the `details` column of the `sys.subtasks` system catalog table.
failed	rebuild_not_possible	The number of missing or damaged segments exceeds the parity width, and the data cannot be recovered currently.	If missing segments are available on an offline drive, you can attempt another rebuild task when that drive is made available to the system. Otherwise, you cannot recover the data.
failed	rebuild_no_space	There is not enough space available to rebuild the segment.	You can complete the rebuild by truncating other data to free up space.
failed	failed_on_node	The cluster lost its connection to the node before the rebuild was completed.	This is a transient error. You can retry the rebuild task.
failed	error	An unexpected internal error occurred.	Review error message details using the `rolehostd` logs. For details, see Log Monitoring.

CREATE TASK Erasure Coding

​Data Segment Statuses

​Recovery Considerations

​Checking for Abnormal Segments

​Starting a Segment Rebuild Task

​Advanced Rebuild Commands

​Checking Rebuild Task Status

​Related Links