Monitor Data Pipelines

You can monitor data pipelines through a combination of:

This tutorial describes how to use these system catalog objects to monitor your pipelines:

A complete reference of system catalog tables and views is available in the Data Pipelines System Catalog.

After you create a pipeline, two convenient views are most useful for observing the pipeline. At a glance, you can view the configuration of your pipeline in information_schema.pipelines and the status information in information_schema.pipeline_status.

Access these views conveniently with the SHOW command.

For a quick look at the progress of a pipeline, the SHOW PIPELINE_STATUS command provides:

To join these two views to other catalog tables, use the pipeline_name to join to the sys.pipelines table and then join tables using the pipeline_id.

Beyond the information schema views, many catalog tables offer details about your pipelines.

While your pipeline is running, the pipeline generates events in the sys.pipeline_events system catalog table to mark significant checkpoints when something has occurred. You can use this table to observe progress and look for detailed messages about changes in the pipeline.

For example, the sys.pipeline_events table captures events when a pipeline is created, started, stopped, fails, and completed. This table also captures relevant system events such as rebalancing Kafka partitions or retrying a process due to a transient failure.

Pipeline events include messages from many different tasks. These tasks are the background processes that execute across different Loader Nodes during pipeline operation.

For file-based loads, the sys.pipeline_files system catalog table contains one row for each file, which contains the status of the file and other file metadata such as filename, creation or modified timestamps, and the file size. The pipeline updates this list of files when it is started and maintains status information as it processes files. You can use this list to understand which files are loading, which succeeded, and which failed.

For Kafka partition-based loads, the sys.pipeline_partitions system catalog table contains one row for each partition, which contains the current offsets and record counts. You can use this information to understand how partitions are distributed across Loader Nodes and also to observe lag on each topic and partition.

Pipeline performance metrics are captured in the sys.pipeline_metrics system catalog table. This table contains samples of the metrics over time. The System collects samples regularly. You can query the samples using standard SQL queries to inspect the behavior of a single pipeline, a single task on a Loader Node, or across all pipelines. The sys.pipeline_metrics_info system catalog table provides metadata that explains the individual metric types.

See Discover Insights from Data Pipeline System Catalog Tables for a detailed description of how to use sys.pipeline_metrics and sys.pipeline_metric_info to monitor the performance of your pipelines. For the system catalog table definitions, see System Catalog.

In the same way that the Ocient System monitors system performance using Statistics Monitoring endpoints with operational monitoring tools, data pipelines expose an API that allows operators to capture performance metrics over time.

You query the metrics using an HTTP endpoint on each Loader Node located at <loader_ip_address_or_hostname>:8090/metrics/v1.

For example, you can retrieve metrics from a Loader Node by executing this command from a shell on the node or by running a similar request from a monitoring agent on the node.

Each metric is represented as a JSON object that contains:

Metrics can be incremental or instantaneous. Incremental metrics accumulate value over time, so they never decrease in value. Instantaneous metrics capture the current value, so they can decrease.

Similar to the sys.pipeline_metrics system catalog table, the Ocient System scopes metrics to a Loader Node, a pipeline task, or a partition. You can determine the scope of a metric based on the presence of the pipeline_name, pipeline_name_external, and partition keys or reference the details in Pipeline Metrics Details.

Examples

This JSON structure contains a metric scoped to a Loader Node.

This JSON structure contains a metric scoped to a pipeline.

This JSON structure contains a metric scoped to a partition.

Data Pipelines System Catalog

System Catalog