Prometheus Metrics Endpoint - Ocient Documentation

The OcientAIQ™ Unified Data Platform exposes a Prometheus®-compatible /metrics endpoint on each node that serves system performance metrics in the OpenMetrics text format. Use this endpoint to integrate with Prometheus, Grafana®, Datadog®, and other monitoring tools that support Prometheus scraping. The /metrics endpoint is available on the same REST interface (port 9090) as the legacy /v1/stats JSON endpoint.

Access the Endpoint

Query the endpoint using curl or any HTTP client.

CURL

curl http://oc1-lts0:9090/metrics

The response uses the Prometheus text exposition format. Output

Prometheus

# HELP jemalloc_stats_bytes jemalloc: memory statistics
# TYPE jemalloc_stats_bytes gauge
# UNIT jemalloc_stats_bytes bytes
jemalloc_stats_bytes{type="active"} 1073741824
jemalloc_stats_bytes{type="allocated"} 1048576000
jemalloc_stats_bytes{type="resident"} 1610612736
# HELP local_storage_service_free_space_bytes Free space on storage device in bytes
# TYPE local_storage_service_free_space_bytes gauge
# UNIT local_storage_service_free_space_bytes bytes
local_storage_service_free_space_bytes{device="SDM00001A616"} 356599726080
# HELP cmdcomp_queries Number of queries by state
# TYPE cmdcomp_queries gauge
cmdcomp_queries{status="execution"} 5
cmdcomp_queries{status="parsing"} 2
cmdcomp_queries{status="queue"} 12

Query Parameters

The /metrics endpoint supports these query parameters to filter the response.

Parameter	Type	Default	Description
`level`	string	`info`	Filter metrics by visibility level with these supported values: `info`, `debug`, or `default`.
`filter`	string	None	Regular expression pattern to match metric names. The endpoint returns only metrics whose names match the pattern.

Filter by Level

Return only production-level (info) metrics. The Ocient® system exports these metrics to Datadog for production monitoring.

CURL

curl "http://oc1-lts0:9090/metrics?level=info"

Include debug-level metrics for troubleshooting.

CURL

curl "http://oc1-lts0:9090/metrics?level=debug"

Filter by Name

Use the filter parameter with a regular expression to return only metrics whose names match the pattern. Narrow the response to a specific functional area instead of retrieving all available metrics. Return only query-processing metrics by matching the cmdcomp prefix.

CURL

curl -G --data-urlencode "filter=^cmdcomp" http://oc1-lts0:9090/metrics

Return all storage-related metrics.

CURL

curl -G --data-urlencode "filter=^local_storage_service" http://oc1-lts0:9090/metrics

Combine both parameters to return only info-level ingestion metrics.

CURL

curl -G --data-urlencode "filter=^stream_loader" "http://oc1-lts0:9090/metrics?level=info"

Metric Types

Each metric has a type that determines how you interpret and aggregate it.

Type	Description	Example
`counter`	A monotonically increasing value that resets only on process restart. Use `rate()` or `increase()` in queries.	`stream_loader_api_push_rows_requests_total`
`gauge`	A value that can increase or decrease. Represents a current measurement.	`cmdcomp_queries{status="execution"}`

Metric Units

When a metric has a defined unit, the system appends the unit name as a suffix to the metric name using Prometheus conventions.

Unit	Suffix	Description
Bytes	`_bytes`	Memory, disk space, and data transfer sizes.
Seconds	`_seconds`	Durations and time measurements.
Percent	`_percent`	Ratios expressed as percentages (0–100).
None	No suffix	Dimensionless counts or ratios.

Metric Labels

Labels provide dimensional metadata that distinguishes different instances of the same metric. For example, cmdcomp_queries uses a status label to differentiate query states.

Prometheus

cmdcomp_queries{status="execution"} 5
cmdcomp_queries{status="parsing"} 2
cmdcomp_queries{status="queue"} 12

Common labels include these categories.

Label	Description	Example Values
`device`	Storage device identifier	Device Universally Unique IDentifier (UUID)
`protocol`	Internal protocol name	`storage_cluster`, `vm`, or `health`
`status`	Operation or query state	`success`, `error`, or `execution`
`type`	Metric subtype or category	`active`, `allocated`, or `page`
`stage`	Pipeline processing stage	`batch_preparation` or `segment_transfer`
`action`	Protocol action name	`on_get_segment_data` or `activate_segment`

Metric Categories

This table summarizes the categories of metrics available with the /metrics endpoint.

Category	Metric Name Prefix	Description
Query processing	`cmdcomp_*`	Query states, metadata cache operations, SQL Node load, and server connections.
Storage devices	`local_storage_service_*`	Disk free space, total space, NVMe SMART health attributes, I/O timeouts, and device status.
Ingestion	`stream_loader_*`	Push rows API request counts, data durability, pipeline stage operations, and durations.
Memory	`jemalloc_stats_`, `memory_heap_`, `memory_huge_*`	Memory allocator statistics and per-subsystem memory usage.
Query execution	`vm_*`	Active queries, datablock router throughput, scheduler state, and dispatch queue events.
Virtual segments	`virtual_segment_`, `virtual_read_cache_`	Segment rebuild activity, read cache hit rates, and block counts.
Storage cluster	`storage_cluster_*`	Segment counts, OSN lifecycle, and cluster data statistics.
Protocol timing	`protocol_time_`, `protocol_actions_`	Internal protocol dispatch delays and action durations.
Raft consensus	`protocol_raft_`, `raft_engine_`	Raft log sizes, snapshot sizes, participant state, and leader method timing.
Disk spill	`disk_based_operator_instance_*`	Bytes spilled to disk during query execution.
System configuration	`system_configuration_store_*`	Configuration refresh durations and operation counts.
Health	`operator_summary_oldest_age_`, `healthProtocolInstance_`	Oldest running operator age and distributed task counts.
TKT pipeline	`tkt_`, `partition_provider_`	Pipeline operator I/O buffer stalls and emits rows.

Key Metrics Reference

These metric families capture monitoring system health, query performance, ingestion throughput, and storage capacity.

Query States and Connections

These metrics provide visibility into the query lifecycle and client connectivity on each SQL Node.

`cmdcomp_queries`

The number of queries currently in each processing state on this node. Use this metric to monitor query queue depth, identify bottlenecks in the compilation pipeline, and detect query pile-ups.

Attribute	Value
Type	`gauge`
Unit	None (count)
Labels	`status`

Label values for status.

Value	Description
`queue`	Queries waiting to begin processing.
`parsing`	Queries that the database parses from SQL text.
`compilation`	Queries that the database compiles into execution plans.
`optimization`	Queries the query planner optimizes.
`validation`	Queries undergoing validation before execution.
`vm_initialization`	Queries initializing the virtual machine execution state.
`execution`	Queries actively executing.

This example alerts when the database queues more than 50 queries.

PromQL

cmdcomp_queries{status="queue"} > 50

`cmdcomp_server_connections`

The number of active client connections to the command compiler server on this node. A sudden drop can indicate a network issue, whereas a sustained increase can indicate connection leaks.

Attribute	Value
Type	`gauge`
Unit	None (count)
Labels	None

`cmdcomp_sql_node_load`

A composite metric representing the total SQL Node load across all nodes. Use this metric to assess whether the query workload is balanced across the cluster.

Attribute	Value
Type	`gauge`
Unit	None
Labels	None

`cmdcomp_metadata_cache_operations`

Counts of metadata cache operations by result type. A low hit rate (high misses relative to requests) might indicate cache pressure or frequent schema changes.

Attribute	Value
Type	`counter`
Unit	None (count)
Labels	`type`

This table describes label values for type.

Value	Description
`hit`	Metadata found without recomputation (cache hit).
`misses`	Metadata fetched or recomputed (cache miss).
`requests`	Total requests to the metadata cache.

This example calculates the metadata cache hit rate over a 5-minute period.

PromQL

rate(cmdcomp_metadata_cache_operations{type="hit"}[5m])
  /
rate(cmdcomp_metadata_cache_operations{type="requests"}[5m])

Ingestion Metrics

These metrics track the data ingestion pipeline from API request receipt through segment generation and transfer.

`stream_loader_api_push_rows_requests`

Total count of push-row API requests that the Loader Node receives by outcome. Use this metric to monitor ingestion request volume and error rates.

Attribute	Value
Type	`counter`
Unit	None (count)
Labels	`status`

This table describes label values for status.

Value	Description
None	Total requests (all outcomes).
`success`	Requests that completed successfully.
`error`	Requests that failed.
`rate_limit_exceeded`	Requests rejected due to rate limiting.

This example calculates the error rate as a percentage over a 5-minute period.

PromQL

rate(stream_loader_api_push_rows_requests{status="error"}[5m])
  /
rate(stream_loader_api_push_rows_requests[5m]) * 100

`stream_loader_api_row_count`

Total number of rows pushed using the API. Use this metric along with the stream_loader_api_push_rows_requests metric to calculate the average batch size.

Attribute	Value
Type	`counter`
Unit	None (count)
Labels	None

`stream_loader_data_durable_bytes`

Bytes that the Loader Node has made durable (persisted) by the storage tier.

Attribute	Value
Type	`gauge`
Unit	Bytes
Labels	`type`

This table describes the label values for type.

Value	Description
`api`	Durable bytes received from the API.
`page`	Durable bytes written to pages.

`stream_loader_data_row_count`

Current number of rows in the Loader Node by durability status.

Attribute	Value
Type	`gauge`
Unit	None (count)
Labels	`durability`

This table describes label values for durability.

Value	Description
`durable`	Rows persisted to storage.
`non_durable`	Rows still in-memory awaiting persistence.

`stream_loader_operations_count`

Counts of pipeline stage operations by stage and outcome. Use this metric to identify the stage in the ingestion pipeline that experiences the most errors or throughput.

Attribute	Value
Type	`counter`
Unit	None (count)
Labels	`stage` or `status`

This table describes label values for stage.

Value	Description
`batch_preparation`	Prepare raw data into batches.
`segment_generation`	Generate storage segments from batches.
`segment_grouping`	Group segments for transfer.
`segment_partitioning`	Partition segments across the cluster.
`segment_transfer`	Transfer segments to storage nodes.

This table describes label values for status.

Value	Description
None	Total operations.
`success`	Successful completions.
`error`	Failed operations.

`stream_loader_operations_duration`

Most recent duration, in seconds, of each pipeline stage. Use this metric to identify slow stages in the ingestion pipeline.

Attribute	Value
Type	`gauge`
Unit	Seconds
Labels	`stage`

Disk Health and Space

These metrics monitor storage device capacity, health, and performance on each Foundation Node.

`local_storage_service_free_space_bytes`

Free space, in bytes, on each storage device. Monitor this value to detect nodes approaching full capacity.

Attribute	Value
Type	`gauge`
Unit	Bytes
Labels	`device`

This example finds devices with less than 10% free space.

PromQL

local_storage_service_space_free_pct < 10

`local_storage_service_total_space_bytes`

Total capacity, in bytes, of each storage device.

Attribute	Value
Type	`gauge`
Unit	Bytes
Labels	`device`

`local_storage_service_space_free_pct`

Percentage of free space on each storage device (0–100).

Attribute	Value
Type	`gauge`
Unit	Percent
Labels	`device`

`local_storage_service_io_timeouts`

Count of NVMe I/O timeouts by the opcode category. Any non-zero value warrants investigation as it might indicate drive degradation.

Attribute	Value
Type	`counter`
Unit	None (count)
Labels	`opcode`

This table describes label values for opcode.

Value	Description
`admin`	Administrator command timeouts.
`read`	Read command timeouts.
`write`	Write command timeouts.
`dsm`	Data set management (TRIM or deallocate) timeouts.
`other`	Other `opcode` timeouts.

`local_storage_service_device_endurance`

Estimated percentage of device life used (0–100). As this value approaches 100, schedule the drive for replacement.

Attribute	Value
Type	`gauge`
Unit	Percent
Labels	`device`

`local_storage_service_available_spare_pct`

Available spare capacity on the NVMe device, normalized to a percentage. When this value drops below the threshold (see the local_storage_service_available_spare_threshold_pct metric), the device controller asserts a warning.

Attribute	Value
Type	`gauge`
Unit	Percent
Labels	`device`

`local_storage_service_crc_errors`

Cumulative count of CRC errors detected by the device. A rising count indicates potential data integrity issues on the link between the host and drive.

Attribute	Value
Type	`counter`
Unit	None (count)
Labels	`device`

Active Queries and Operator Age

These metrics provide high-level visibility into query execution activity and help detect stuck or long-running operations.

`vm_active_queries_count`

Total number of queries currently executing on the virtual machine of this node. Use this metric to gauge overall query concurrency and correlate with resource consumption.

Attribute	Value
Type	`gauge`
Unit	None (count)
Labels	None

This example alerts when active queries exceed the expected concurrency limit.

PromQL

vm_active_queries_count > 200

`operator_summary_oldest_age_seconds`

Age, in seconds, of the oldest currently running operator instance on this node. A large value might indicate a stuck query or an operation waiting on an unavailable resource.

Attribute	Value
Type	`gauge`
Unit	Seconds
Labels	None

This example alerts when any operator has been running for more than one hour.

PromQL

operator_summary_oldest_age_seconds > 3600

`disk_based_operator_instance_bytes_spilled`

Total bytes that query operators have spilled to disk when the database exceeds in-memory budgets. A high spill volume indicates that queries are exceeding available memory and resorting to disk-based processing, which is significantly slower.

Attribute	Value
Type	`counter`
Unit	Bytes
Labels	`compression`

This table describes label values for compression.

Value	Description
`compressed`	Bytes written to disk after compression.
`uncompressed`	Logical bytes spilled before compression.

This example monitors the spill rate in bytes per second.

PromQL

rate(disk_based_operator_instance_bytes_spilled{compression="uncompressed"}[5m])

Configure Prometheus Scraping

If you run a Prometheus server, you can configure it to periodically scrape the /metrics endpoint on each Ocient node. Prometheus stores the scraped metrics in its time series database, making them available for PromQL® queries and Grafana dashboards. Add a scrape job to your prometheus.yml configuration file. List every Ocient node as a target and set metrics_path to /metrics. The params section passes query parameters, such as level, with each scrape request.

prometheus.yml

scrape_configs:
  - job_name: 'ocient'
    scrape_interval: 30s
    params:
      level: ['info']
    static_configs:
      - targets:
          - 'oc1-lts0:9090'
          - 'oc1-lts1:9090'
          - 'oc1-sql0:9090'
    metrics_path: '/metrics'

After you save and reload the Prometheus configuration, the targets appear on the Prometheus Status > Targets page. Prometheus begins collecting metrics at the configured scrape_interval interval.

Configure the Datadog Agent

If you use Datadog instead of Prometheus, Ocient provides an official Datadog integration that scrapes the same /metrics endpoint and forwards metrics to the Datadog cloud platform. For details, see Datadog Integration.

Example Queries

These example PromQL queries demonstrate common monitoring use cases. You can execute them in the Prometheus expression browser, in a Grafana dashboard panel, or with any tool that supports PromQL. Disk Space Utilization Calculate the percentage of free disk space on each storage device. A declining value indicates that the node is running low on storage and might need capacity planning or data retention adjustments.

PromQL

local_storage_service_free_space_bytes / local_storage_service_total_space_bytes * 100

Query Queue Depth Return the current number of queries waiting in the queue. A consistently high value might indicate that the system is under heavy query load or that additional SQL Nodes are needed to handle the workload.

PromQL

cmdcomp_queries{status="queue"}

Loader Node Error Rate Calculate the ratio of failed push-row API requests to total requests over the last 5 minutes. The rate() function converts the raw counter into a per-second rate. A rising error rate might indicate issues with the data source, schema mismatches, or pipeline configuration errors.

PromQL

rate(stream_loader_api_push_rows_requests_total{status="error"}[5m])
  /
rate(stream_loader_api_push_rows_requests_total[5m])

Memory Allocator Pressure Compare the amount of memory actively in use by jemalloc to the total memory mapped from the operating system. A ratio approaching 1.0 indicates that most mapped memory is in active use, which might signal memory pressure on the node.

PromQL

jemalloc_stats_bytes{type="active"} / jemalloc_stats_bytes{type="mapped"}

Comparison with the Legacy `/v1/stats` Endpoint

This table compares the legacy /v1/stats endpoint features to the Prometheus metrics.

Feature	`/v1/stats` (Legacy)	`/metrics` (Prometheus)
Format	JSON	Prometheus text exposition
Naming	Dot-delimited (e.g., `localStorageService.device.spaceFree`)	Underscore-delimited with unit suffix (e.g., `local_storage_service_free_space_bytes`)
Labels	Encoded in metric name (e.g., `device[UUID]`)	Prometheus labels (e.g., `{device="UUID"}`)
Filtering	`filter` query parameter (regex)	`filter` and `level` query parameters
Tooling	Custom JSON parsers or Telegraf™	Prometheus, Grafana, Datadog, or any OpenMetrics client
Level filtering	Not supported	`level=info\|debug\|default`

Both endpoints remain available on port 9090 and serve the same underlying metric data.

​Access the Endpoint

​Query Parameters

​Filter by Level

​Filter by Name

​Metric Types

​Metric Units

​Metric Labels

​Metric Categories

​Key Metrics Reference

​Query States and Connections

​cmdcomp_queries

​cmdcomp_server_connections

​cmdcomp_sql_node_load

​cmdcomp_metadata_cache_operations

​Ingestion Metrics

​stream_loader_api_push_rows_requests

​stream_loader_api_row_count

​stream_loader_data_durable_bytes

​stream_loader_data_row_count

​stream_loader_operations_count

​stream_loader_operations_duration

​Disk Health and Space

​local_storage_service_free_space_bytes

​local_storage_service_total_space_bytes

​local_storage_service_space_free_pct

​local_storage_service_io_timeouts

​local_storage_service_device_endurance

​local_storage_service_available_spare_pct

​local_storage_service_crc_errors

​Active Queries and Operator Age

​vm_active_queries_count

​operator_summary_oldest_age_seconds

​disk_based_operator_instance_bytes_spilled

​Configure Prometheus Scraping

​Configure the Datadog Agent

​Example Queries

​Comparison with the Legacy /v1/stats Endpoint

​Related Links

Access the Endpoint

Query Parameters

Filter by Level

Filter by Name

Metric Types

Metric Units

Metric Labels

Metric Categories

Key Metrics Reference

Query States and Connections

`cmdcomp_queries`

`cmdcomp_server_connections`

`cmdcomp_sql_node_load`

`cmdcomp_metadata_cache_operations`

Ingestion Metrics

`stream_loader_api_push_rows_requests`

`stream_loader_api_row_count`

`stream_loader_data_durable_bytes`

`stream_loader_data_row_count`

`stream_loader_operations_count`

`stream_loader_operations_duration`

Disk Health and Space

`local_storage_service_free_space_bytes`

`local_storage_service_total_space_bytes`

`local_storage_service_space_free_pct`

`local_storage_service_io_timeouts`

`local_storage_service_device_endurance`

`local_storage_service_available_spare_pct`

`local_storage_service_crc_errors`

Active Queries and Operator Age

`vm_active_queries_count`

`operator_summary_oldest_age_seconds`

`disk_based_operator_instance_bytes_spilled`

Configure Prometheus Scraping

Configure the Datadog Agent

Example Queries

Comparison with the Legacy `/v1/stats` Endpoint

Related Links