Skip to main content
Data in an System is erasure coded to provide resilience to disk and Foundation Node failures. The resilience of a system depends on the parity width of the storage space, which represents the number of associated disks (or nodes) that can fail without interruption of service or data loss. For details about storage spaces and parity width, see Configure Storage Spaces.

​
Data Segment Statuses

You can check the status of your system segment groups by querying the sys.segment_groups system catalog table. For details, see System Catalog. This table describes the states for data segments.
StatusDescriptionRecovery Process
INTACTThe normal and operable status.No recovery needed.
DAMAGEDThe segment failed a checksum, meaning the segment data is corrupted and unusable.If you have sufficient parity width, you can recover a damaged segment by invoking a rebuild segment task.
MISSINGThe segment is on a node or disk that is currently offline. If the node or disk rejoins the cluster, it can transition to the INTACT status.When a disk or Foundation Node is permanently removed, you can perform a rebuild task to recover the data. This requires sufficient parity width.
REBUILDINGThe segment is in recovery after damage or missing segment data.Recovery is already in progress.

​
Recovery Considerations

If a segment has the DAMAGED or MISSING status, queries can proceed by reconstructing the data on demand using the remaining erasure-coded data in the segment group. Having a non-INTACT status means that input or output (I/O) performance is significantly reduced. To restore full performance, you need to run a segment rebuild task. Segment rebuilding is not automatic. The system administrator must manually invoke it. A segment rebuild fails, and data is lost completely if the number of segments with the DAMAGED status in a segment group exceeds the parity width of the storage space. To avoid data loss:
  • Provision the parity width of the storage space at or above the number of expected concurrent node failures.
  • Rebuild damaged or permanently missing segments as soon as possible.

​
Checking for Abnormal Segments

You can find any segments that need a rebuild by querying for segment groups with an abnormal status. Examples Finding Faulty Segment Groups This example query finds any segment groups with the DAMAGED or MISSING status.
SQL
SELECT * FROM sys.segment_groups WHERE status IN ('DAMAGED', 'MISSING');
Output
Text
|         "id"         |              "cluster_id"              | "segment_type" | "status" | "primary_owner" |              "loader_id"               |               "table_id"               |               "scope_id"               | "block_size" | "begin_time" | "end_time" | "coding_algorithm" | "coding_block_size" | "coding_threshold" | "coding_width" | "replication" | "parity_cycle" |     "created_time"      | "rolehostd_version" |               "commit_hash"                |    "timestamp"    | "build_user" | "depth" | "removal_type" |
|----------------------|----------------------------------------|----------------|----------|-----------------|----------------------------------------|----------------------------------------|----------------------------------------|--------------|--------------|------------|--------------------|---------------------|--------------------|----------------|---------------|----------------|-------------------------|---------------------|--------------------------------------------|-------------------|--------------|---------|----------------|
| "720576045199095878" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT    | DAMAGED  |                 | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | bd18d33b-aae9-4be2-a2f4-76ecdc34c2ba   | "8bc119d2-875a-47e4-b016-87fde83e77d6" |         4096 |            0 |          1 | XOR_PARITY         |                4096 |                  2 |              3 |             1 |              1 | 2025-02-26 20:23:34.274 | "25.0.0"            | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" |              |       0 | NOT_REMOVED    |
| "716072445571725367" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT    | DAMAGED  |                 | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | "9fe79009-ee75-4429-b29b-3a614a166751" | a8c805d0-2144-46a7-8374-52cb88d64244   |         4096 |            0 |          1 | XOR_PARITY         |                4096 |                  2 |              3 |             1 |              1 | 2025-02-26 20:23:19.447 | "25.0.0"            | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" |              |       0 | NOT_REMOVED    |
Finding Clusters and Nodes with Faulty Segment Groups Inspect the count of damaged groups by cluster and node.
SQL
SELECT
  c.name AS cluster_name,
  n.name AS node_name,
  g.status AS segment_group_status,
  seg.status AS segment_status,
  seg.kind,
  COUNT(*) AS segment_count
FROM
  sys.segment_groups g
LEFT JOIN
  sys.clusters c
  ON c.id = g.cluster_id
LEFT JOIN
  sys.stored_segments seg
  ON seg.segment_group_id = g.id
LEFT JOIN
  sys.nodes n
  ON n.id = seg.node_id
WHERE
  g.status <> 'INTACT'
  AND (seg.status <> 'INTACT' OR seg.status IS NULL)
  GROUP BY 1,2,3,4,5
  ORDER BY 1,2,3,4,5;
Output
Text
|   "cluster_name"   | "node_name" | "segment_group_status" | "segment_status" | "kind"  | "segment_count" |
|--------------------|-------------|------------------------|------------------|---------|-----------------|
| foundation_cluster | foundation0 | DAMAGED                |                  | VIRTUAL |               2 |

​
Starting a Segment Rebuild Task

A user with System Administrator privileges can start a segment rebuild using the CREATE TASK TYPE REBUILD SQL statement. You cannot cancel a segment rebuild task after it is started. Most commonly, a rebuild task repairs all damaged or missing segments across the system. Example Create a rebuild task.
SQL
CREATE TASK TYPE REBUILD;
The system can continue to perform queries while rebuilding segments, but the process can impact I/O performance.

​
Advanced Rebuild Commands

Rebuild tasks can also execute on specific Foundation Nodes or clusters. For information on fine-tuning rebuild tasks, see CREATE TASK.

​
Checking Rebuild Task Status

Monitor the status of current and past segment rebuild tasks from the sys.subtasks system catalog table. For details, see System Catalog.
SQL
SELECT * FROM sys.subtasks WHERE task_type = 'rebuild';
This table describes the statuses for a rebuild task.
StatusStatus DetailDescriptionNext Steps
completeno_workThe segment groups were already available and healthy. No rebuilding was needed.None
completecompleteRebuild completed successfully.None
runningrebuild_in_progressRebuild is in process.You can monitor the progress of the rebuild task by checking the JSON dictionary in the details column of the sys.subtasks system catalog table.
failedrebuild_not_possibleThe number of missing or damaged segments exceeds the parity width, and the data cannot be recovered currently.If missing segments are available on an offline drive, you can attempt another rebuild task when that drive is made available to the system.
Otherwise, you cannot recover the data.
failedrebuild_no_spaceThere is not enough space available to rebuild the segment.You can complete the rebuild by truncating other data to free up space.
failedfailed_on_nodeThe cluster lost its connection to the node before the rebuild was completed.This is a transient error. You can retry the rebuild task.
failederrorAn unexpected internal error occurred.Review error message details using the rolehostd logs. For details, see Log Monitoring.
CREATE TASK Erasure Coding
Last modified on May 27, 2026