# Aggregate Functions Source: https://docs.ocient.com/aggregate-functions Reference for Ocient SQL aggregate functions, including SUM, AVG, COUNT, MIN, MAX, and approximate aggregates for computing values across rows in queries. The has aggregated and sorted aggregate functions. Aggregate and sorted aggregate functions compute a single result from a group of rows. You can use the `DISTINCT` keyword with any aggregation function, such as `COUNT(DISTINCT col)`. Whereas sorted aggregate functions use the standard ORDER BY syntax to dictate the ordering of the aggregation. To sort the elements of an array, use the [ARRAY\_SORT](/transform-data-in-data-pipelines#array-data-transformation-functions) function during data load. ## Aggregate Functions Supported input types vary by function. In general, numeric types include `TINYINT`, `SMALLINT`, `INT`, `BIGINT`, `FLOAT`, `DOUBLE`, and `DECIMAL`. Various examples on this page use these tables. ```sql SQL theme={null} CREATE TABLE sample_data (val INT, label VARCHAR(10)); INSERT INTO sample_data (val, label) VALUES (2, 'alpha'), (4, 'beta'), (4, 'alpha'), (5, NULL), (7, 'gamma'), (8, 'alpha'), (12, 'beta'), (15, 'gamma'); ``` ```sql SQL theme={null} CREATE TABLE predictions (actual INT, predicted INT); INSERT INTO predictions (actual, predicted) VALUES (10, 12), (20, 18), (30, 33), (40, 37), (50, 52), (60, 58); ``` For simple functions, examples use the `sys.dummy` virtual table. For details, see [Generate Tables Using sys.dummy](/generate-tables-using-sys-dummy). ### ACCURACY\_SCORE Returns the fraction of predictions that match the actual class labels. This function calculates the ratio of correctly predicted rows to the total number of rows. ```sql SQL theme={null} ACCURACY_SCORE(y, y_hat) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ------------------- | -------------------------- | | `y` | All types supported | The actual class label. | | `y_hat` | All types supported | The predicted class label. | The return type is `DOUBLE`. The result represents the proportion of correct predictions, ranging from 0 to 1. **Examples** **Calculate the Accuracy Score for Matched Values** This query returns the accuracy score when all predictions match the actual values. ```sql SQL theme={null} SELECT ACCURACY_SCORE(c1, c1) FROM sys.dummy10; ``` Output: `1` **Calculate the Accuracy Score with Comparison Against Squared Values** This query returns the accuracy score when comparing actual values against their squared values. ```sql SQL theme={null} SELECT ACCURACY_SCORE(c1, c1 * c1) FROM sys.dummy10; ``` Output: `0.1` ### ANY\_VALUE Returns an arbitrary non-`NULL` value from the input column. The function returns `NULL` only if all rows in the column are `NULL`. The `ANY_VALUE` function does not support window aggregation. **Syntax** ```sql SQL theme={null} ANY_VALUE(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ------------- | -------------------------------------------------------------- | | `col` | Any type | The column from which to return an arbitrary non-`NULL` value. | **Example** ```sql SQL theme={null} SELECT ANY_VALUE(label) AS sample_label FROM sample_data; ``` Output: `alpha` The returned value is non-deterministic. The database can return any non-`NULL` value from the column. ### APPROX\_COUNT\_DISTINCT Returns an approximate count of distinct values in the column using the HyperLogLog algorithm, with a 95% confidence interval that the result is within 4.5% of the exact count. **Syntax** ```sql SQL theme={null} APPROX_COUNT_DISTINCT(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------- | | `col` | Any non-interval type | The column for the approximation of the distinct count. | **Example** ```sql SQL theme={null} SELECT APPROX_COUNT_DISTINCT(val) AS approx_distinct FROM sample_data; ``` Output: `7` The `sample_data` table has 8 rows, but there are only 7 distinct values in the `val` column (the value `4` appears twice). ### APPROX\_SUM Computes a sum using a faster, non-deterministic ordering for floating-point columns. This summation can lead to minor differences in the result on the order of the machine epsilon. For integral column types, the function uses the standard `SUM` algorithm. **Syntax** ```sql SQL theme={null} APPROX_SUM(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | --------------------------------------- | ------------------ | | `col` | `FLOAT`, `DOUBLE`, or any integral type | The column to sum. | For integral inputs, the `APPROX_SUM` function uses the standard `SUM` algorithm and returns the same result. The performance benefit applies to large floating-point data sets. **Example** ```sql SQL theme={null} SELECT APPROX_SUM(c1) AS approx_total FROM sys.dummy5; ``` Output: `15` ### AVG Computes the arithmetic mean over the set of values. **Syntax** ```sql SQL theme={null} AVG(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `col` | Any numeric type or `MATRIX` | The column to average.

Integral and `FLOAT` inputs return `DOUBLE`.

`DOUBLE`, `DECIMAL`, and `MATRIX` inputs preserve their type. | **Example** ```sql SQL theme={null} SELECT AVG(c1) AS avg_val FROM sys.dummy5; ``` Output: `3.0` ### COEFFICIENT\_OF\_DETERMINATION Computes the coefficient of determination (R²) between actual and predicted values. Primarily used to evaluate the performance of machine learning regression models, R² measures the proportion of variance in the actual values that the predicted values explain. A value of `1.0` indicates a perfect fit, while `0.0` indicates that the predictions explain none of the variance. This function returns `NULL` when the total sum of squares is zero (i.e., all actual values are identical). **Syntax** ```sql SQL theme={null} COEFFICIENT_OF_DETERMINATION(actual, predicted) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------- | ------------------------------- | | `actual` | Any numeric type | The column of observed values. | | `predicted` | Any numeric type | The column of predicted values. | **Example** ```sql SQL theme={null} SELECT COEFFICIENT_OF_DETERMINATION(actual, predicted) AS r_squared FROM predictions; ``` Output: `0.980571428571429` ### CONFUSION\_MATRIX Returns a structured representation of the counts for every combination of actual and predicted class labels relative to a specified positive class. This function aggregates predictions into a 2x2 matrix in the format `[[TP, FP], [FN, TN]]` for binary classification analysis. ```sql SQL theme={null} CONFUSION_MATRIX(y, y_hat, positive_class) ``` | **Argument** | **Data Type** | **Description** | | ---------------- | ------------------- | --------------------------------------------------------------------------------------- | | `y` | All types supported | The actual class label. | | `y_hat` | All types supported | The predicted class label. | | `positive_class` | All types supported | The class label to treat as the positive class. Must match the type of `y` and `y_hat`. | The return type is `TUPLE`. Each entry in the result contains the count of rows for a specific combination of actual and predicted class values, organized as true positives, false positives, false negatives, and true negatives. **Examples** **Calculate the Confusion Matrix with Matched Values** This query returns the confusion matrix when all predictions match the actual values. ```sql SQL theme={null} SELECT CONFUSION_MATRIX( CASE WHEN c1 > 5 THEN 1 ELSE 0 END, CASE WHEN c1 > 5 THEN 1 ELSE 0 END, 1 ) FROM sys.dummy10; ``` Output: `[[5, 0], [0, 5]]` **Calculate the Confusion Matrix with Mismatched Classifiers** This query returns the confusion matrix for two mismatched binary classifiers, where actual positives are rows with `c1 > 5` and predicted positives are rows with `c1 < 3`. ```sql SQL theme={null} SELECT CONFUSION_MATRIX( CASE WHEN c1 > 5 THEN 1 ELSE 0 END, CASE WHEN c1 < 3 THEN 1 ELSE 0 END, 1 ) FROM sys.dummy10; ``` Output: `[[0, 2], [5, 3]]` ### CORR Alias for [CORRELATIONP](#correlationp). ### CORRELATION Alias for [CORRELATIONP](#correlationp). ### CORRELATIONP Computes the population Pearson correlation coefficient between two columns. Returns `DOUBLE`, or `DECIMAL` if both inputs are `DECIMAL`-compatible. Alias for `CORRELATION` and `CORR`. **Syntax** ```sql SQL theme={null} CORRELATIONP(col1, col2) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------- | ------------------ | | `col1` | Any numeric type | The first column. | | `col2` | Any numeric type | The second column. | **Example** ```sql SQL theme={null} SELECT CORRELATIONP(actual, predicted) AS corr_val FROM predictions; ``` Output: `0.9904654955172436` ### COUNT Returns the number of rows in the set where values in `col` are not `NULL`. When you use the `COUNT(*)` SQL statement, the function returns the total number of rows. The return type is `BIGINT`. **Syntax** ```sql SQL theme={null} COUNT(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | --------------------- | ----------------------------------------------------------------------------------------- | | `col` | Any non-interval type | The column for counting non-`NULL` values.

Use `*` to count all rows instead. | **Examples** **Count All Rows** ```sql SQL theme={null} SELECT COUNT(*) AS total_rows FROM sample_data; ``` Output: `8` **Count Non-NULL Values** The `label` column has one `NULL` row. ```sql SQL theme={null} SELECT COUNT(label) AS rows_with_label FROM sample_data; ``` Output: `7` ### COVAR\_POP Alias for [COVARIANCEP](#covariancep). ### COVAR\_SAMP Alias for [COVARIANCE](#covariance). ### COVARIANCE Computes the sample covariance between two columns. Returns `DOUBLE` by default, or `DECIMAL` if both inputs are `DECIMAL`-compatible. `MATRIX` inputs preserve their type. Alias for `COVAR_SAMP`. **Syntax** ```sql SQL theme={null} COVARIANCE(col1, col2) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ------------------------------------- | ------------------ | | `col1` | Any numeric type or `MATRIX` (square) | The first column. | | `col2` | Any numeric type or `MATRIX` (square) | The second column. | Matrix arguments must have matching dimensions. **Example** ```sql SQL theme={null} SELECT COVARIANCE(actual, predicted) AS cov_samp FROM predictions; ``` Output: `336.0` ### COVARIANCEP Computes the population covariance between two columns. Returns `DOUBLE` by default, or `DECIMAL` if both inputs are compatible with `DECIMAL`. `MATRIX` inputs preserve their type. Alias for `COVAR_POP`. **Syntax** ```sql SQL theme={null} COVARIANCEP(col1, col2) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ------------------------------------- | ------------------ | | `col1` | Any numeric type or `MATRIX` (square) | The first column. | | `col2` | Any numeric type or `MATRIX` (square) | The second column. | Matrix arguments must have matching dimensions. **Example** ```sql SQL theme={null} SELECT COVARIANCEP(actual, predicted) AS cov_pop FROM predictions; ``` Output: `280.0` ### F1\_SCORE Returns the harmonic mean of precision and recall for a specified positive class. This function calculates `2 * (Precision * Recall) / (Precision + Recall)` and provides a single score that balances both precision and recall. The function returns `NULL` when both precision and recall are 0. ```sql SQL theme={null} F1_SCORE(y, y_hat, positive_class) ``` | **Argument** | **Data Type** | **Description** | | ---------------- | ------------------- | --------------------------------------------------------------------------------------- | | `y` | All types supported | The actual class label. | | `y_hat` | All types supported | The predicted class label. | | `positive_class` | All types supported | The class label to treat as the positive class. Must match the type of `y` and `y_hat`. | The return type is `DOUBLE`. The result ranges from 0 to 1, where 1 represents perfect precision and recall. **Examples** **Calculate the F1 Score with Matching Values** This query returns the F1 score when all predictions match the actual values. ```sql SQL theme={null} SELECT F1_SCORE( CASE WHEN c1 > 5 THEN 1 ELSE 0 END, CASE WHEN c1 > 5 THEN 1 ELSE 0 END, 1 ) FROM sys.dummy10; ``` Output: `1` **Calculate the F1 Score for Mismatched Classifiers** This query returns `NULL` because both precision and recall are 0 for the mismatched classifiers. ```sql SQL theme={null} SELECT F1_SCORE( CASE WHEN c1 > 5 THEN 1 ELSE 0 END, CASE WHEN c1 < 3 THEN 1 ELSE 0 END, 1 ) FROM sys.dummy10; ``` Output: `NULL` ### KURTOSIS Computes the sample excess kurtosis over the set of values. Kurtosis measures the difference between the tails of a distribution and the tails of a normal distribution. Returns `DOUBLE`, or `DECIMAL` if the input is `DECIMAL`. **Syntax** ```sql SQL theme={null} KURTOSIS(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------- | --------------------------------------------- | | `col` | Any numeric type | The column for computing the sample kurtosis. | **Example** ```sql SQL theme={null} SELECT KURTOSIS(val) AS kurt_val FROM sample_data; ``` Output: `4.763725276787391` ### KURTOSISP Computes the population kurtosis over the set of values. Returns `DOUBLE`, or `DECIMAL` if the input is `DECIMAL`. **Syntax** ```sql SQL theme={null} KURTOSISP(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------- | --------------------------------------------- | | `col` | Any numeric type | The column for computing population kurtosis. | **Example** ```sql SQL theme={null} SELECT KURTOSISP(val) AS kurtp_val FROM sample_data; ``` Output: `2.268440607993995` ### MAX Returns the maximum value in the specified column. The return type matches the input. **Syntax** ```sql SQL theme={null} MAX(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | --------------------- | ------------------------------------------- | | `col` | Any non-interval type | The column for returning the maximum value. | **Example** ```sql SQL theme={null} SELECT MAX(c1) AS max_val FROM sys.dummy5; ``` Output: `5` ### MEAN\_ABSOLUTE\_ERROR Returns the mean absolute error (MAE) between actual and predicted values. This function calculates the average of the absolute differences between each pair of actual and predicted values, treating all errors equally regardless of direction. ```sql SQL theme={null} MEAN_ABSOLUTE_ERROR(y, y_hat) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ------------- | -------------------- | | `y` | NUMERIC | The actual value. | | `y_hat` | NUMERIC | The predicted value. | The return type is `DOUBLE`. The result is always zero or positive. A value of 0 indicates perfect predictions. Examples **Calculate the MAE for Matched Values** This query returns the mean absolute error when all predictions match the actual values. ```sql SQL theme={null} SELECT MEAN_ABSOLUTE_ERROR(c1, c1) FROM sys.dummy10; ``` Output: `0` **Calculate the MAE with Comparison Against Squared Values** This query returns the mean absolute error when comparing actual values against their squared values. ```sql SQL theme={null} SELECT MEAN_ABSOLUTE_ERROR(c1, c1 * c1) FROM sys.dummy10; ``` Output: `33.0` ### MEAN\_ABSOLUTE\_PERCENTAGE\_ERROR Returns the mean absolute percentage error (MAPE) between actual and predicted values. This function calculates the average of the absolute percentage differences using the formula `Average( AbsoluteValue( (y - y_hat) / y ) ) * 100`, which expresses the error relative to the actual values. ```sql SQL theme={null} MEAN_ABSOLUTE_PERCENTAGE_ERROR(y, y_hat) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ------------- | ----------------------------------------------------------------------------------------------------- | | `y` | NUMERIC | The actual value. Must not be 0, because this value is the denominator in the percentage calculation. | | `y_hat` | NUMERIC | The predicted value. | The return type is `DOUBLE`. The result is a percentage value where 0 indicates perfect predictions. Examples **Calculate the MAPE for Matched Values** This query returns the mean absolute percentage error when all predictions match the actual values. ```sql SQL theme={null} SELECT MEAN_ABSOLUTE_PERCENTAGE_ERROR(c1, c1) FROM sys.dummy10; ``` Output: `0` **Calculate the MAPE with Comparison Against Squared Values** This query returns the mean absolute percentage error when comparing actual values against their squared values. ```sql SQL theme={null} SELECT MEAN_ABSOLUTE_PERCENTAGE_ERROR(c1, c1 * c1) FROM sys.dummy10; ``` Output: `450.0` ### MIN Returns the minimum value in the specified column. The return type matches the input. **Syntax** ```sql SQL theme={null} MIN(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | --------------------- | ------------------------------------------- | | `col` | Any non-interval type | The column for returning the minimum value. | **Example** ```sql SQL theme={null} SELECT MIN(c1) AS min_val FROM sys.dummy5; ``` Output: `1` ### PRECISION\_SCORE Returns the precision score for a specified positive class. This function calculates the ratio of true positives to the total number of predicted positives using the formula `True Positives / (True Positives + False Positives)`. Of all the predictions for the specified class, this score measures how many are correct. ```sql SQL theme={null} PRECISION_SCORE(y, y_hat, positive_class) ``` | **Argument** | **Data Type** | **Description** | | ---------------- | ------------------- | --------------------------------------------------------------------------------------- | | `y` | All types supported | The actual class label. | | `y_hat` | All types supported | The predicted class label. | | `positive_class` | All types supported | The class label to treat as the positive class. Must match the type of `y` and `y_hat`. | The return type is `DOUBLE`. The result ranges from 0 to 1, where 1 indicates that every prediction for the positive class is correct. **Examples** **Calculate the Precision Score for Matched Values** This query returns the precision score when all predictions match the actual values. ```sql SQL theme={null} SELECT PRECISION_SCORE( CASE WHEN c1 > 5 THEN 1 ELSE 0 END, CASE WHEN c1 > 5 THEN 1 ELSE 0 END, 1 ) FROM sys.dummy10; ``` Output: `1` **Calculate the Precision Score for Mismatched Classifiers** This query returns the precision score for two mismatched binary classifiers, where none of the predicted positives are actual positives. ```sql SQL theme={null} SELECT PRECISION_SCORE( CASE WHEN c1 > 5 THEN 1 ELSE 0 END, CASE WHEN c1 < 3 THEN 1 ELSE 0 END, 1 ) FROM sys.dummy10; ``` Output: `0.0` ### PRODUCT Computes the product of all values in the column. **Syntax** ```sql SQL theme={null} PRODUCT(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | --------------------------------------------------------------------- | -------------------------------------------------------- | | `col` | `FLOAT`, `DOUBLE`, `DECIMAL`, any integral type, or `MATRIX` (square) | The column to multiply. Integral inputs return `BIGINT`. | Matrix arguments must have matching dimensions. **Example** ```sql SQL theme={null} SELECT PRODUCT(c1) AS factorial_5 FROM sys.dummy5; ``` Output: `120` ### RECALL\_SCORE Returns the recall score for a specified positive class. This function calculates the ratio of true positives to the total number of actual positives using the formula `True Positives / (True Positives + False Negatives)`. Of all the actual instances of the specified class, this score measures how many the model correctly identifies. ```sql SQL theme={null} RECALL_SCORE(y, y_hat, positive_class) ``` | **Argument** | **Data Type** | **Description** | | ---------------- | ------------------- | --------------------------------------------------------------------------------------- | | `y` | All types supported | The actual class label. | | `y_hat` | All types supported | The predicted class label. | | `positive_class` | All types supported | The class label to treat as the positive class. Must match the type of `y` and `y_hat`. | The return type is `DOUBLE`. The result ranges from 0 to 1, where 1 indicates that every actual instance of the positive class is correctly identified. **Examples** **Calculate the Recall Score for Matched Values** This query returns the recall score when all predictions match the actual values. ```sql SQL theme={null} SELECT RECALL_SCORE( CASE WHEN c1 > 5 THEN 1 ELSE 0 END, CASE WHEN c1 > 5 THEN 1 ELSE 0 END, 1 ) FROM sys.dummy10; ``` Output: `1` **Calculate the Recall Score for Mismatched Classifiers** This query returns the recall score for two mismatched binary classifiers, where none of the actual positives are correctly predicted. ```sql SQL theme={null} SELECT RECALL_SCORE( CASE WHEN c1 > 5 THEN 1 ELSE 0 END, CASE WHEN c1 < 3 THEN 1 ELSE 0 END, 1 ) FROM sys.dummy10; ``` Output: `0.0` ### ROC\_AUC\_SCORE Returns the area under the receiver operating characteristic (ROC) curve (AUC). This function measures the ability of a binary classification model to distinguish between classes. A score of 1.0 represents a perfect classifier, while a score of 0.5 represents performance no better than random guessing. ```sql SQL theme={null} ROC_AUC_SCORE(y_true, y_score, positive_class) ``` | **Argument** | **Data Type** | **Description** | | ---------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `y_true` | BOOLEAN or INTEGER | The true binary class label, where the values represent two classes (for example, 0 or 1). | | `y_score` | NUMERIC | The predicted probability or confidence score for the positive class. A score of 1.0 is a perfect classifier, while 0.5 is no better than random guessing. | | `positive_class` | | The class label to treat as the positive class. Must match the type of `y_true`. | The return type is `DOUBLE`. The result ranges from 0 to 1. **Example** This query returns the AUC score for a classifier in which higher probability scores correspond to actual positives, indicating perfect class separation. ```sql SQL theme={null} SELECT ROC_AUC_SCORE( CASE WHEN c1 > 5 THEN 1 ELSE 0 END, DOUBLE(c1) / 10, 1 ) FROM sys.dummy10; ``` Output: `1.0` ### SKEW Computes the sample skewness over the set of values. Skewness measures the asymmetry of a distribution. Returns `DOUBLE`, or `DECIMAL` if the input is `DECIMAL`. **Syntax** ```sql SQL theme={null} SKEW(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------- | --------------------------------------------- | | `col` | Any numeric type | The column for computing the sample skewness. | **Example** ```sql SQL theme={null} SELECT SKEW(val) AS skew_val FROM sample_data; ``` Output: `0.8804164883619399` ### SKEWP Computes the population skewness over the set of values. Returns `DOUBLE`, or `DECIMAL` if the input is `DECIMAL`. **Syntax** ```sql SQL theme={null} SKEWP(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------- | ------------------------------------------------- | | `col` | Any numeric type | The column for computing the population skewness. | **Example** ```sql SQL theme={null} SELECT SKEWP(val) AS skewp_val FROM sample_data; ``` Output: `0.7059036122393627` ### STDEV Computes the sample standard deviation over the set of values. Returns `DOUBLE`, or `DECIMAL` if the input is `DECIMAL`. Alias for `STDDEV` and `STDDEV_SAMP`. **Syntax** ```sql SQL theme={null} STDEV(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------- | ------------------------------------------------------- | | `col` | Any numeric type | The column for computing the sample standard deviation. | **Example** ```sql SQL theme={null} SELECT STDEV(val) AS stdev_val FROM sample_data; ``` Output: `4.421942042651783` ### STDDEV Alias for [STDEV](#stdev). ### STDDEV\_POP Alias for [STDEVP](#stdevp). ### STDDEV\_SAMP Alias for [STDEV](#stdev). ### STDEVP Computes the population standard deviation over the set of values. Returns `DOUBLE`, or `DECIMAL` if the input is `DECIMAL`. Alias for `STDDEV_POP`. **Syntax** ```sql SQL theme={null} STDEVP(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------- | ----------------------------------------------------------- | | `col` | Any numeric type | The column for computing the population standard deviation. | **Example** ```sql SQL theme={null} SELECT STDEVP(val) AS stdevp_val FROM sample_data; ``` Output: `4.136348026943574` ### SUM Computes the sum of all values in the column. **Syntax** ```sql SQL theme={null} SUM(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `col` | Any numeric type or `MATRIX` | The column to sum.

Integral inputs return `BIGINT`.
d
`FLOAT` inputs return `FLOAT`.

`DOUBLE`, `DECIMAL`, and `MATRIX` inputs preserve their type. | **Example** ```sql SQL theme={null} SELECT SUM(c1) AS total FROM sys.dummy5; ``` Output: `15` ### VAR\_POP Alias for [VARIANCEP](#variancep). ### VAR\_SAMP Alias for [VARIANCE](#variance). ### VARIANCE Computes the sample variance over the set of values. Returns `DOUBLE`, or `DECIMAL` if the input is `DECIMAL`. `MATRIX` inputs preserve their type and must be square. Alias for `VAR_SAMP`. **Syntax** ```sql SQL theme={null} VARIANCE(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ------------------------------------- | --------------------------------------------- | | `col` | Any numeric type or `MATRIX` (square) | The column for computing the sample variance. | Matrix arguments must have matching dimensions. **Example** ```sql SQL theme={null} SELECT VARIANCE(val) AS var_val FROM sample_data; ``` Output: `19.553571428571427` ### VARIANCEP Computes the population variance over the set of values. Returns `DOUBLE`, or `DECIMAL` if the input is `DECIMAL`. `MATRIX` inputs preserve their type and must be square. Alias for `VAR_POP`. **Syntax** ```sql SQL theme={null} VARIANCEP(col) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ------------------------------------- | --------------------------------------------- | | `col` | Any numeric type or `MATRIX` (square) | The column for computing population variance. | **Example** ```sql SQL theme={null} SELECT VARIANCEP(val) AS varp_val FROM sample_data; ``` Output: `17.109375` ## Sorted Aggregate Functions The general syntax for sorted aggregate functions adds the DISTINCT and ORDER BY keywords in the function invocation. ```sql SQL theme={null} AGGREGATE([DISTINCT] arg1, arg2, ... [ORDER BY ...]) ``` ### ARRAY\_AGG Returns an array containing every row from the expression. **Syntax** ```sql SQL theme={null} ARRAY_AGG(expr) ``` | **Argument** | **Data Type** | **Description** | | ------------ | ---------------- | ----------------------------------------- | | `expr` | Any numeric type | Expression for aggregation into an array. | **Example** Aggregate ten rows into an array in descending order. ```sql SQL theme={null} SELECT ARRAY_AGG(c1 ORDER BY c1 DESC) FROM sys.dummy10; ``` Output: `[10,9,8,7,6,5,4,3,2,1]` ### ARRAY\_CONCAT\_AGG Returns an array that concatenates arrays across rows. The input argument is a SQL expression. The element type must be consistent across rows. The function ignores NULL inputs and returns NULL only when all inputs are NULL. The `DISTINCT` keyword removes duplicate arrays. To remove duplicate elements from an array, use the `ARRAY_DISTINCT` function during load. The `ORDER BY` syntax controls the order of concatenation across rows. To sort elements in the array, use the `ARRAY_SORT` function during load. **Syntax** ```sql SQL theme={null} ARRAY_CONCAT_AGG(expr) ``` | **Argument** | **Data Type** | **Description** | | ------------ | -------------- | ----------------------------------------- | | `expr` | SQL expression | Expression for concatenation across rows. | **Example** Create the `data_type_example` table with a column that stores arrays of integers. ```sql SQL theme={null} CREATE TABLE data_type_example (col_int_array INT[] NOT NULL DEFAULT 'INT[0,1,2,3]'); ``` Insert one array with values `1`, `2`, and `3` into the table. ```sql SQL theme={null} INSERT INTO data_type_example SELECT ARRAY[1,2,3]; ``` Insert the second array with values `4`, `5`, and `6` into the table. ```sql SQL theme={null} INSERT INTO data_type_example SELECT ARRAY[4,5,6]; ``` Concatenate the two arrays into one array in ascending order. ```sql SQL theme={null} SELECT ARRAY_CONCAT_AGG(col_int_array ORDER BY col_int_array ASC) FROM data_type_example; ``` Output: `[1,2,3,4,5,6]` ### STRING\_AGG Returns a string concatenated from every row of the expression. **Syntax** ```sql SQL theme={null} ARRAY_CONCAT_AGG(expr, delimiter) ``` | **Argument** | **Data Type** | **Description** | | ------------ | -------------- | ----------------------------------------- | | `expr` | SQL expression | Expression for concatenation across rows. | | `delimiter` | | Optional.
| **Examples** **Concatenate Rows as a String** Concatenate ten rows into a string in descending order. Cast integers into strings using the `CHAR` casting function. ```sql SQL theme={null} SELECT STRING_AGG(char(c1) ORDER BY c1 DESC) FROM sys.dummy10; ``` Output: `"10987654321"` **Concatenate Rows as a String with a Delimiter** Concatenate ten rows into a string in ascending order. Cast integers into strings using the `CHAR` casting function. Use the `|` delimiter. ```sql SQL theme={null} SELECT STRING_AGG(char(c1), '|' ORDER BY c1 ASC) FROM sys.dummy10; ``` Output: `"1|2|3|4|5|6|7|8|9|10"` ## Related Links [Math Functions and Operators](/math-functions-and-operators) [Query Ocient](/query-ocient) # Air-Gapped Environment with an Ocient System Source: https://docs.ocient.com/air-gapped-environment-with-an-ocient-system Deploy and operate an Ocient System in an air-gapped environment to protect sensitive data, with no special configuration required for full isolation. An air gap is a security measure that physically isolates a computer or network from any other network, including the internet. This isolation removes wired or wireless connections between the isolated system and any external and potentially unsecured networks. Organizations that handle highly sensitive information, such as government agencies, financial institutions, and critical infrastructure operators, use air-gapped environments to protect against various security incidents. supports using an Ocient System in an air-gapped environment without any special configuration. Here, you can find the general workflow for using an Ocient System in an air-gapped environment and the required installation steps. ## Work with an Ocient System in an Air-Gapped Environment This workflow shows the high-level steps to set up an Ocient System in an air-gapped environment. Ensure that your drive firmware and operating system is up to date. Install and configure an Ocient System in a connected environment where you can download the required files from the internet. Disassemble the cluster. Pack and deliver the hardware to your secured location. Reassign a new IP address. Reassign a new hostname. Bootstrap the system again. ## Install an Ocient System in an Air-Gapped Environment Follow these steps to install the Ocient System in an air-gapped environment. These steps assume that you have met the system requirements for installation. #### Bootstrap the Node Connect to the initial SQL Node with the username and password of your server and the IP address of your node. This example connects as the administrator `admin` to the IP Address `10.10.10.10`. ```shell Shell theme={null} ssh admin@10.10.10.10 ``` Use your preferred text editor with `sudo` to create the `/var/opt/ocient/bootstrap.conf` file as root with these contents. `/var/opt/ocient/bootstrap.conf` **example** ```yaml YAML theme={null} initialSystem: true ``` Start the database. ```shell Shell theme={null} sudo systemctl start rolehostd ``` Verify that the node and the service is active by executing this status command. ```shell Shell theme={null} systemctl status rolehostd rolehostd.service - Rolehostd daemon startup Loaded: loaded (/etc/systemd/system/rolehostd.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2022-01-26 23:31:36 UTC; 7s ago ... ``` If the `rolehostd` service is running, you can also check the Ocient logs on your node. Search and ensure there are no \[ERROR] log messages. ```shell Shell theme={null} tail -f /var/opt/ocient/log/rolehostd.log ``` #### Verify Connection to the SQL Node At this point, you have a running database with a single node. You should be able to connect to the database using JDBC or `pyocient` and execute commands. Every new system starts with a `system` database. To connect to a new system, use the username and password configured in the `bootstrap.conf` file or the username `admin@system` and password `admin` if none were provided. For example, assume your node named `sql0` has an IP address of `10.10.0.1`. Use the JDBC driver CLI to connect with this connection string. ```shell Shell theme={null} connect to jdbc:ocient://10.10.0.1:4050/system user "myuser@system" using "mypassword"; ``` To see the roles running on the single node, execute this query. ```sql SQL theme={null} select name, operational_status, software_version, array_agg(service_role_type) from sys.node_status as ns left join sys.nodes as n on ns.node_id = id left join sys.service_roles as sr on sr.node_id = n.id group by name, operational_status,software_version; ``` The initial node should be listed as running the `sql`, `admin`, `health`, and `operatorvm` roles. If all of these roles are present and the node is Active, you can proceed to the next step to bootstrap the remaining nodes. Performing the bootstrapping process on the remaining nodes is identical on all nodes. The remaining nodes can be bootstrapped in any order. On each node, log in using SSH and use your text editor with `sudo` to create the file `/var/opt/ocient/bootstrap.conf` that contains this text by replacing `` with the IP address of the initial node you created in Step 1. `/var/opt/ocient/bootstrap.conf` **example** ```yaml YAML theme={null} adminHost: ``` `` is the DNS name or IP address of the initial node. You can obtain the IP Address of the initial node by executing `ifconfig` on that node. If the password for the system administrator has changed, set the correct username `adminUserName` and password `adminPassword` in the bootstrap configuration file `bootstrap.conf`. On each node, start the database. ```shell Shell theme={null} sudo systemctl start rolehostd ``` When you replace Foundation Nodes, the Ocient System removes the prior node after the creation of the new node. Some queries of the system catalog tables might not return results until the prior node is removed. At this point, all the remaining nodes are not configured with any roles. After all nodes have been started, you should see them when you execute this query with only the `health` role listed. This query uses the `sys.node_status`, `sys.nodes`, and `sys.service_roles` system catalog tables to retrieve node information for the node name, operation status, version, and all service role types. The query uses the ARRAY\_AGG function to retrieve the service role type for all rows. ```sql SQL theme={null} SELECT name, operational_status, software_version, array_agg(service_role_type) FROM sys.node_status AS ns LEFT JOIN sys.nodes AS n ON ns.node_id = id LEFT JOIN sys.service_roles AS sr ON sr.node_id = n.id GROUP BY name, operational_status,software_version; ``` ## Remove a Cluster in an Air-Gapped Environment Use this information and workflow to remove a cluster by shutting down all Ocient processes and erasing the drives and data. The system administrator must download the `sedutil-cli` utility used in this workflow. You can classify the drives in an Ocient System in two categories based on their usage. * Operating System (OS) drives: These drives contain the installation of the operating system and software, including the . A system might have a single physical OS drive or an OS installed on a RAID disk created by using more than one drive. The OS drives on nodes with the administrator role (Metadata and possibly SQL Nodes) also store configuration information related to the Ocient System that can include: * Node names * Node IP addresses * User data mapping information for compressed columns * Encryption keys for the data drives (unless the keys are under the control of an external key management system) * Data drives: The data drives are present in all types of nodes except the node running only the administrator role. On the Foundation Node, the data drives store tables. On SQL Nodes, the data drives store transient query information. And, on Loader Nodes, the data drives store transient loading information. The data drives in the Ocient System are exclusively NVMe drives. The OS drives can be NVMe or SSD. These drives can support The Computing Group (TCG) Opal Specification or not support it. The type of drive (Opal-supported or not) determines how the system removes all data on the drive. Follow these steps to remove a cluster for Opal-supported drives. To remove data for Opal-supported drives, follow these steps. These steps irreversibly remove the data. Follow these steps after you ensure that you no longer need the applicable data from the system. Stop the Ocient processes running on the nodes using these commands. For Loader Nodes, use this command. The command combines two commands that stop all processes related to loading. ```shell Shell theme={null} sudo systemctl disable lat && sudo systemctl stop lat ``` For all types of nodes, use this command. The command combines two commands where the first command stops processing on all nodes and the second command stops the main Ocient System process. ```shell Shell theme={null} sudo systemctl disable rolehostd && sudo systemctl kill -s SIGKILL rolehostd ``` Bind the data drives to an NVMe driver so that the drives become visible to the OS. ```shell Shell theme={null} sudo /opt/ocient/scripts/nvme-driver-util.sh bind-nvme ``` Run this command for each data drive. The impacted drives are the drives displayed as their association change from `uio` drivers to `nvme` drivers. You can see the `nvme` drive association by running the `/opt/ocient/scripts/nvme-driver-util.sh` script. All data drives are erased. You can move drives to another node with the `sedutil-cli` utility if the nodes are already powered off and you cannot power them on. Use the same command to erase the drive: `sedutil-cli -n --revertTPer admin /dev/nvmeXn1`. ```shell Shell theme={null} sedutil-cli -n --revertTPer admin /dev/nvmeXn1 ``` You cannot securely erase the OS drives while the OS is running. You can either use the secure erase facility from BIOS or move drives to an external host with a utility approved by your organization for the remove operation. ## Related Links [Install an Ocient System](/install-an-ocient-system) [Ocient System Bootstrapping](/ocient-system-bootstrapping) [Set Up System Monitoring with the TIG Stack and Kapacitor](/set-up-system-monitoring-with-the-tig-stack-and-kapacitor) # Alphabetical SQL Functions List Source: https://docs.ocient.com/alphabetical-sql-functions-list Alphabetical reference of all SQL functions in Ocient, including math, string, date, conversion, geospatial, machine learning, and aggregates. | **Function Name** | **Category Name** | **Function Description** | | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [ABS](/math-functions-and-operators#abs) | Math Functions | Returns the absolute value of a specified floating-point number. | | [ABS](/matrix-functions-and-operators#matrix-functions) | Matrix Functions | Returns the magnitude of one-dimensional matrix or vector. | | [ACOS](/math-functions-and-operators#acos) | Math Functions | Returns the inverse cosine of a specified floating-point number. | | [ACOSH](/math-functions-and-operators#acosh) | Math Functions | Returns the hyperbolic arc-cosine of a specified floating-point number. | | [ACCURACY\_SCORE](/aggregate-functions#accuracy_score) | Aggregate Functions | Returns the fraction of predictions that match the actual class labels. | | [ADD\_MONTHS](/date-and-time-functions#add_months) | Date and Time Functions | Adds the specified number of months to the date. | | [ANY\_VALUE](/aggregate-functions#any_value) | Aggregate Functions | Returns an arbitrary, non-NULL value from the input column. | | [APPROX\_COUNT\_DISTINCT](/aggregate-functions#approx_count_distinct) | Aggregate Functions | Approximate distinct count by using hyper-log-log (95% confidence interval that the value is within 4.5%). | | [APPROX\_SUM](/aggregate-functions#approx_sum) | Aggregate Functions | Allows the aggregation engine to use a faster, non-deterministic ordering to summate floating-point columns. | | [ARRAY\[\]](/array-functions-and-operators) | Array Functions | -compliant constructor. The type of the array is deduced from the elements. | | [ARRAY\_AGG](/aggregate-functions#array_agg) | Aggregate Functions | Returns an array containing every row from the expression. | | [ARRAY\_APPEND](/array-functions-and-operators) | Array Functions | Add value to the back of an array. | | [ARRAY\_ARGMAX](/array-functions-and-operators) | Array Functions | Returns the corresponding argmax of the array as a BIGINT index. | | [ARRAY\_ARGMIN](/array-functions-and-operators) | Array Functions | Returns the corresponding argmin of the array as a BIGINT index. | | [ARRAY\_CAT](/array-functions-and-operators) | Array Functions | Concatenate 2 arrays into a new one. | | [ARRAY\_CAT\_DISTINCT](/array-functions-and-operators#array_cat_distinct) | Array Functions | Concatenates two or more arrays in the order of the input arguments. | | [ARRAY\_CAP](/transform-data-in-data-pipelines#array-data-transformation-functions) | Special Data Pipeline Transformation Functions | Restrict the length of an array to a maximum number of elements. | | [ARRAY\_COMPACT](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Removes NULL values from the array. | | [ARRAY\_CONCAT\_AGG](/aggregate-functions#array_concat_agg) | Aggregate Functions | Returns an array that concatenates arrays across rows. | | [ARRAY\_CONTAINS](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Returns `true` if the array contains the specified value. | | [ARRAY\_DISTINCT](/array-functions-and-operators#array_distinct) | Array Functions | Removes duplicates from an array while preserving the first occurrence order. | | [ARRAY\_LENGTH](/array-functions-and-operators) | Array Functions | Return the number of elements of a given array. | | [ARRAY\_MAX](/array-functions-and-operators) | Array Functions | Returns the corresponding maximum of the array. | | [ARRAY\_MIN](/array-functions-and-operators) | Array Functions | Returns the corresponding minimum of the array. | | [ARRAY\_POSITION](/array-functions-and-operators) | Array Functions | Returns the position of the first matching scalar in the array. | | [ARRAY\_POSITIONS](/array-functions-and-operators) | Array Functions | Returns all elements stored in the right array and returns their respective positions in the left array. | | [ARRAY\_PREPEND](/array-functions-and-operators) | Array Functions | Add value to the front of an array. | | [ARRAY\_REMOVE](/array-functions-and-operators) | Array Functions | Remove value from the array. | | [ARRAY\_REPLACE](/array-functions-and-operators) | Array Functions | Replace value by another in an array. | | [ARRAY\_SORT](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Sort and return the input array based on the natural ordering of its elements or the specified Lambda function. | | [ARRAY\_SUM](/array-functions-and-operators) | Array Functions | Returns the sum of the array. The array must be one-dimensional and contain numeric values. NULL values do not contribute to the sum. | | [ARRAY\_TO\_STRING](/array-functions-and-operators) | Array Functions | Converts array to a list of elements separated by 'delimiter'. | | [ASCII](/character-and-binary-functions#ascii) | Character and Binary Functions | Returns the ASCII code value of the leftmost character of the character value. | | [ASIN](/math-functions-and-operators#asin) | Math Functions | Returns the inverse sine of a specified floating-point number. | | [ASINH](/math-functions-and-operators#asinh) | Math Functions | Returns the hyperbolic arc-sine of a specified floating-point number. | | [ATAN](/math-functions-and-operators#atan) | Math Functions | Returns the inverse tangent of a specified floating-point number. | | [ATAN2](/math-functions-and-operators#atan2) | Math Functions | Returns the inverse tangent of two numeric, floating-point values. | | [ATANH](/math-functions-and-operators#atanh) | Math Functions | Returns the hyperbolic arc-tangent of a specified floating-point number. | | [AVG](/aggregate-functions#avg) | Aggregate Functions | Average, or arithmetic mean, over the set. | | [BICDF](/math-functions-and-operators#bicdf) | Math Functions | The cumulative distribution function of the standard bivariate normal distribution. | | [BIGINT](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type `bigint`. | | [BINARY](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses a hexadecimal string (such as 0x54ab) to create a binary value. Letters can be of either case. | | [BIPDF](/math-functions-and-operators#bipdf) | Math Functions | The probability density function of the standard bivariate normal distribution. | | [BIT\_LENGTH](/character-and-binary-functions#bit_length) | Character and Binary Functions | Returns the length of the character value in bits. | | [BITAND](/math-functions-and-operators#bitand) | Math Functions | Alias for the BITFUNC syntax `BITFUNC('AND', x, y)`. | | [BITFUNC](/math-functions-and-operators#bitfunc) | Math Functions | Performs a variety of bit operations. Can be any of these string literals: `'AND'`, `'OR'`, or `'XOR'`. | | [BITNOT](/math-functions-and-operators#bitnot) | Math Functions | Returns the bitwise negation of integral\_x. | | [BITOR](/math-functions-and-operators#bitor) | Math Functions | Alias for the BITFUNC syntax BITFUNC('OR', x, y). | | [BITXOR](/math-functions-and-operators#bitxor) | Math Functions | Alias for the BITFUNC syntax BITFUNC('XOR', x, y). | | [BOOLAND](/math-functions-and-operators#booland) | Math Functions | Alias for the BOOLFUNC syntax BOOLFUNC('AND', x, y). | | [BOOLEAN](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses a string to create a Boolean value. The string must contain either true or false. It is case-insensitive. | | [BOOLFUNC](/math-functions-and-operators#boolfunc) | Math Functions | Performs a Boolean logical evaluation on arguments x and y. Can be any of these string literals: `'AND'`, `'OR'`, or `'XOR'`. | | [BOOLNOT](/math-functions-and-operators#boolnot) | Math Functions | Returns the logical negation of the BOOLFUNC function. | | [BOOLOR](/math-functions-and-operators#boolor) | Math Functions | Alias for the BOOLFUNC syntax BOOLFUNC('OR', x, y). | | [BOOLXOR](/math-functions-and-operators#boolxor) | Math Functions | Alias for the BOOLFUNC syntax BOOLFUNC('XOR', x, y). | | [BTRIM](/character-and-binary-functions#btrim) | Character and Binary Functions | Alias for TRIM. | | [BYTE](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type byte. | | [CANCEL](/query-management#cancel) | Query Management | Cancels a running query based on its specific query identifier. | | [CASE](/other-functions-and-expressions#case)
| Conditional Functions | CASE operates similarly to conditional scripting in other programming languages, allowing it to function like an if / then / else statement or as a switch statement. | | [CASE WHEN](/transform-data-in-data-pipelines#special-data-pipeline-transformation-functions)

| Special Data Pipeline Transformation Functions | Returns the `result` value based on whether an expression is `true`. | | [CAST\_TO\_ARRAY](/array-functions-and-operators) | Array Functions | Casts the elements of the array to another type. | | [CAST\_TO\_TUPLE](/tuple-functions-and-operators) | Tuple Functions | Converts a tuple into another tuple of a different type. | | [CBRT](/math-functions-and-operators#cbrt) | Math Functions | Returns the cube root of the numeric value x. | | [CDF](/math-functions-and-operators#cdf) | Math Functions | The cumulative distribution function of the standard normal distribution. Returns the probability that a random sample is less than or equal to the specified value. | | [CEIL](/math-functions-and-operators#ceil) | Math Functions | Returns the nearest integer greater than or equal to x. | | [CEILING](/math-functions-and-operators#ceiling) | Math Functions | Alias for CEIL. | | [CENTURY](/date-and-time-functions#century) | Date and Time Functions | Returns the number of centuries. | | [CHAR](/array-functions-and-operators) | Array Functions | Converts array to its string representation. | | [CHAR](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Creates a string version of the numeric value. | | [CHAR](/tuple-functions-and-operators) | Tuple Functions | Converts a tuple to its string representation. | | [CHAR\_LENGTH](/character-and-binary-functions#char_length) | Character and Binary Functions | Alias for LENGTH. | | [CHARACTER\_LENGTH](/character-and-binary-functions#character_length) | Character and Binary Functions | Alias for LENGTH. | | [CHR](/character-and-binary-functions#chr) | Character and Binary Functions | Converts an integer value to a string. | | [COALESCE](/other-functions-and-expressions#coalesce) | Conditional Functions | Evaluates to the first argument that is not NULL, or NULL if all arguments are NULL. | | [COEFFICIENT\_OF\_DETERMINATION](/aggregate-functions#coefficient_of_determination) | Aggregate Functions | Computes the coefficient of determination (R²) between actual and predicted values. | | [COMMIT](/other-functions-and-expressions#commit) | System Functions | Returns the most recent commit hash of the database to which the client is currently connected. | | [CONCAT](/character-and-binary-functions#concat) | Character and Binary Functions | Concatenates two values, which must both be either binary, hash, or string data types. This function is equivalent to the \|\| operator. | | [CONFUSION\_MATRIX](/aggregate-functions#confusion_matrix) | Aggregate Functions | Returns a structured representation of the counts for every combination of actual and predicted class labels relative to a specified positive class. | | [CONVERT\_LOCAL\_TIMESTAMP\_TO\_UTC](/time-zone-functions#convert_local_timestamp_to_utc) | Time Zone Functions | The function converts a timestamp in a specified local time zone to the UTC time zone. | | [CONVERT\_UTC\_TIMESTAMP\_TO\_LOCAL](/time-zone-functions#convert_utc_timestamp_to_local) | Time Zone Functions | The function converts a timestamp from the UTC time zone to a specified local time zone. | | [CORR](/aggregate-functions#corr) | Aggregate Functions | Alias for CORRELATION. | | [CORRELATION](/aggregate-functions#correlation) | Aggregate Functions | Sample correlation. | | [CORRELATIONP](/aggregate-functions#correlationp) | Aggregate Functions | Population correlation. | | [COS](/math-functions-and-operators#cos) | Math Functions | Returns the cosine of x. | | [COSH](/math-functions-and-operators#cosh) | Math Functions | Returns the hyperbolic cosine of x. | | [COT](/math-functions-and-operators#cot) | Math Functions | Returns the cotangent of x. | | [COUNT](/aggregate-functions#count) | Aggregate Functions | Number of rows in the set. | | [COVAR\_POP](/aggregate-functions#covar_pop) | Aggregate Functions | Alias for COVARIANCEP. | | [COVAR\_SAMP](/aggregate-functions#covar_samp) | Aggregate Functions | Alias for COVARIANCE. | | [COVARIANCE](/aggregate-functions#covariance) | Aggregate Functions | Sample covariance. | | [COVARIANCEP](/aggregate-functions#covariancep) | Aggregate Functions | Population covariance. | | [CROSS\_ENTROPY\_LOSS](/array-functions-and-operators) | Array Functions | Returns the cross entropy loss of two arrays. | | [CROSS\_ENTROPY\_LOSS](/matrix-functions-and-operators) | Matrix Functions | Returns the cross entropy loss of two one-dimensional matrices or vectors. | | [CUME\_DIST](/window-aggregate-functions#cume_dist) | Window Aggregate Functions | Returns a number 0 \< n 1 and can be used to calculate the percentage of values less than or equal to the current value in the group. | | [CURDATE](/date-and-time-functions#curdate) | Date and Time Functions | Alias for CURRENT\_DATE. | | [CURRENT\_DATABASE](/other-functions-and-expressions#current_database) | System Functions | Alias for DATABASE. | | [CURRENT\_DATE](/date-and-time-functions#current_date) | Date and Time Functions | Returns the current date in the format YYYY-MM-DD. | | [CURRENT\_GROUPS](/other-functions-and-expressions#current_groups) | System Functions | Returns the fully-qualified names of groups in the database. | | [CURRENT\_NODE](/other-functions-and-expressions#current_node) | System Functions | Returns the name of the SQL Node where the current query executes. The name of the node corresponds to the name column in the `sys.nodes` system catalog table. | | [CURRENT\_NODE\_ID](/other-functions-and-expressions#current_node_id) | System Functions | Returns the identifier of the SQL Node where the current query executes. | | [CURRENT\_SCHEMA](/other-functions-and-expressions#current_schema) | System Functions | Returns the name of the current schema. | | [CURRENT\_SESSION\_ID](/other-functions-and-expressions#current_session_id) | System Functions | Returns the Universally Unique IDentifier (UUID) of the current session. | | [CURRENT\_SYSTEM](/other-functions-and-expressions#current_system) | System Functions | Returns the name of the system. | | [CURRENT\_TIME](/date-and-time-functions#current_time) | Date and Time Functions | Returns the current time as a TIME value (e.g., `hh:mm:ss.mm`). | | [CURRENT\_TIMESTAMP](/date-and-time-functions#current_timestamp) | Date and Time Functions | Returns the current date and time as a TIMESTAMP value (e.g., YYYY-MM-DD hh🇲🇲ss.mmm). | | [CURRENT\_USER](/other-functions-and-expressions#current_user) | System Functions | Returns the user for the current connection. | | [DATABASE](/other-functions-and-expressions#database) | System Functions | Returns the name of the database to which the client is currently connected. | | [DATE](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses a string in the form 'YYYY-MM-DD' to create a date. Extra characters are ignored. | | [DATE\_PART](/date-and-time-functions#date_part) | Date and Time Functions | Alias for EXTRACT. | | [DATE\_TRUNC](/date-and-time-functions#date_trunc) | Date and Time Functions | Returns the date or timestamp entered, truncated to the specified precision. | | [DATEADD](/date-and-time-functions#dateadd) | Date and Time Functions | Adds a specified number value (as a signed integer) to a specified date part of an input date value, and then returns that modified value. | | [DATEDIFF](/date-and-time-functions#datediff) | Date and Time Functions | This function returns an INT representing the difference between two date or time values in a specified date or time unit. | | [DAY](/date-and-time-functions#day) | Date and Time Functions | Alias for DAY\_OF\_MONTH. | | [DAY\_OF\_MONTH](/date-and-time-functions#day_of_month) | Date and Time Functions | Extracts the day-of-month portion of a timestamp or date as an integer. | | [DAY\_OF\_WEEK](/date-and-time-functions#day_of_week) | Date and Time Functions | Returns an integer, in the range of 1 to 7, that represents the day of the week. | | [DAY\_OF\_YEAR](/date-and-time-functions#day_of_year) | Date and Time Functions | Returns an integer in the range 1 to 366 that represents the day of the year. | | [DAYS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type days to be used in date calculations. | | [DECADE](/date-and-time-functions#decade) | Date and Time Functions | The decade is the year divided by 10. | | [DECIMAL](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type decimal. | | [DEGREES](/math-functions-and-operators#degrees) | Math Functions | Returns the corresponding angle in degrees for x in radians. | | [DELTA](/window-aggregate-functions#delta) | Window Aggregate Functions | Computes the finite difference between successive values of expression under the specified ordering. This is a backwards difference, which means that, at degree one, the value for a given row is the difference between the value of expression for that row and the previous row. | | [DENSE\_RANK](/window-aggregate-functions#dense_rank) | Window Aggregate Functions | Assigns a number to each row in the result set with equal values having the same number. There will be no gaps between ranks. | | [DERIVATIVE](/window-aggregate-functions#derivative) | Window Aggregate Functions | Computes the difference quotient between successive values of expression with respect to expression2. | | [DET](/matrix-functions-and-operators) | Matrix Functions | Returns the determinant of the matrix as a double. | | [DIV](/math-functions-and-operators#div) | Math Functions | Returns the result of x divided by y. If y is zero, returns NULL. | | [DOT](/matrix-functions-and-operators) | Matrix Functions | Returns dot product of two one-dimensional matrices/vectors. | | [DOUBLE](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type double. | | [DOW](/date-and-time-functions#dow) | Date and Time Functions | Alias for DAY\_OF\_WEEK. | | [DOY](/date-and-time-functions#doy) | Date and Time Functions | Alias for DAY\_OF\_YEAR. | | [EIGEN](/matrix-functions-and-operators) | Matrix Functions | Returns eigenvalues and eigenvalues of a square matrix as a vector of pairs. | | [ELEMENT\_AT](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Returns the element value of the array or tuple at the specified index. | | [ENDSWITH](/character-and-binary-functions#endswith) | Character and Binary Functions | Returns true if x ends with y and false otherwise. | | [EOMONTH](/date-and-time-functions#eomonth) | Date and Time Functions | Returns the last day of the timestamp or date. | | [EPOCH](/date-and-time-functions#epoch) | Date and Time Functions | The number of seconds after 1970-01-01 00:00:00 UTC. | | [ERF](/math-functions-and-operators#erf) | Math Functions | The error function is used for measurements that follow a normal distribution. | | [ERFC](/math-functions-and-operators#erfc) | Math Functions | The complement of the error function. ERFC(x) = 1 - ERF(x). | | [EXP](/math-functions-and-operators#exp) | Math Functions | Returns the exponential of x (e raised to the power of x). | | [EXPLODE\_OUTER](/special-data-pipeline-transformation-functions#explode_outer) | Special Data Pipeline Transformation Functions | Expands a one-dimensional or multidimensional array into its elements with one element per row of output from the system. | | [EXTRACT](/date-and-time-functions#extract) | Date and Time Functions | Extract a component from a timestamp or date. | | [F1\_SCORE](/aggregate-functions#f1_score) | Aggregate Functions | Returns the harmonic mean of precision and recall for a specified positive class. | | [FILTER](/special-data-pipeline-transformation-functions#filter) | Special Data Pipeline Transformation Functions | Filters elements in an array based on the logic in a lambda expression. | | [FIRST\_VALUE](/window-aggregate-functions#first_value) | Window Aggregate Functions | Returns the first value in the ordered result set. | | [FLATTEN](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Transforms an `N`-dimensional array into an `N-1`-dimensional array. | | [FLOAT](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type float. | | [FLOOR](/math-functions-and-operators#floor) | Math Functions | Returns the nearest integer less than or equal to x. | | [FROBENIUS](/matrix-functions-and-operators) | Matrix Functions and Operators | Returns the Frobenius norm of a matrix. | | [GAMMA](/math-functions-and-operators#gamma) | Math Functions | [Gamma function.](https://en.wikipedia.org/wiki/Gamma_function) | | [GREATEST](/other-functions-and-expressions#greatest) | Conditional Functions | Returns the largest non-NULL value of all the arguments, or NULL if all the arguments are NULL. | | [HASH](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Creates a fixed length binary value, with length ``, from a string, i.e. `0x1234abcd`. Zero extended if the string does not have enough bytes, truncated if it has too many. | | [HEXBINX](/math-functions-and-operators#hexbinx) | Math Functions | Returns the x-coordinate of the center of the nearest hexagonal bin to the point (x, y). | | [HEXBINY](/math-functions-and-operators#hexbiny) | Math Functions | Returns the y-coordinate of the center of the nearest hexagonal bin to the point (x, y). | | [HINGE\_LOSS](/array-functions-and-operators) | Array Functions | Returns the hinge loss of two arrays. | | [HINGE\_LOSS](/matrix-functions-and-operators) | Matrix Functions | Returns the hinge loss of two one-dimensional matrices or vectors. | | [HLL\_SKETCH\_CREATE](/hyperloglog-functions#hll_sketch_create) | HyperLogLog Functions | Creates an HLL sketch from the data on a specified aggregated column. Returns a HASH((2^log2k) + 8) data representation of the sketch that you can store in a separate column. | | [HLL\_SKETCH\_GET\_ESTIMATE](/hyperloglog-functions#hll_sketch_get_estimate) | HyperLogLog Functions | The scalar function converts a sketch into a distinct count estimate of a sketch value. Returns the distinct count estimate as a BIGINT. | | [HLL\_SKETCH\_GET\_ESTIMATE\_BOUND](/hyperloglog-functions#hll_sketch_get_estimate_bound) | HyperLogLog Functions | Takes a HLL\_SKETCH column or an integral log2k literal value and returns the resulting bounding 95-percent confidence interval error proportion as a DOUBLE. | | [HLL\_SKETCH\_TO\_STRING](/hyperloglog-functions#hll_sketch_to_string) | HyperLogLog Functions | The HLL\_SKETCH\_TO\_STRING scalar function takes a HLL\_SKETCH column or value and returns a string summary of the sketch. | | [HLL\_SKETCH\_UNION (aggregate function)](/hyperloglog-functions#hll_sketch_union-aggregate-function) | HyperLogLog Functions | Merges multiple sketches in a single column into a unified sketch. All sketches must have the same precision. This function is an aggregate function and operates on a column. | | [HLL\_SKETCH\_UNION (scalar function)](/hyperloglog-functions#hll_sketch_union-scalar-function) | HyperLogLog Functions | Merges two sketches into a new combined sketch. This function is a scalar function and operates row-wise. The scalar function merges two sketch columns with heterogeneous precisions into a sketch with the lower of the two precisions. | | [HOUR](/date-and-time-functions#hour) | Date and Time Functions | Extracts the hour portion of a timestamp as an integer. | | [HOURS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type hours. | | [IDENTITY\_MATRIX](/matrix-functions-and-operators) | Matrix Functions | Returns an identity matrix of the specified dimension. | | [IERF](/math-functions-and-operators#ierf) | Math Functions | The inverse of the ERF error function. | | [IERFC](/math-functions-and-operators#ierfc) | Math Functions | The inverse of the complement of the error function. | | [IF](/transform-data-in-data-pipelines#logical-operations-transformation-functions) | Special Data Pipeline Transformation Functions | Returns `T` if the expression `X` evaluates to `true`, or the function returns `F` if `X` evaluates to `false`. | | [IF\_NULL](/other-functions-and-expressions#if_null) | Conditional Functions | Alias for COALESCE. | | [INITCAP](/character-and-binary-functions#initcap) | Character and Binary Functions | For each word in the provided string, capitalize the first character if it is alphabetic. | | [INSTR](/character-and-binary-functions#instr) | Character and Binary Functions | Returns the index position of the first occurrence where the character value char\_substring appears in the character value char by ignoring case. | | [INTEGER](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type integer. | | [INVERSE](/matrix-functions-and-operators) | Matrix Functions | Returns inverse of a square, invertible matrix. | | [IP](/network-type-functions#ip) | Network Type Functions | Casts an IP data type from an IPV4 data type expression. | | [IPV4](/network-type-functions#ipv4) | Network Type Functions | Casts an IPV4 address from an IPV6 address. | | [ISDATE](/date-and-time-functions#isdate) | Date and Time Functions | Returns TRUE if the input argument can be successfully cast to a date. | | [IS\_IPV4](/network-type-functions#is_ipv4) | Network Type Functions | Tests whether the database can convert the `IP` value to the `IPV4` data type. | | [ISODOW](/date-and-time-functions#isodow) | Date and Time Functions | Extracts the day of the week based on ISO 8601, which ranges from Monday (1) to Sunday (7). | | [JSON\_EXTRACT\_PATH\_TEXT](/character-and-binary-functions#json_extract_path_text) | Character and Binary Functions | Returns the value for the key-value pair referenced by a series of path elements in a JSON string. | | [KILL](/query-management#kill) | Query Management | Cancels a running query based on its specific query identifier. | | [KURTOSIS](/aggregate-functions#kurtosis) | Aggregate Functions | The sample over the set. | | [KURTOSISP](/aggregate-functions#kurtosisp) | Aggregate Functions | The population over the set. | | [LAG](/window-aggregate-functions#lag) | Window Aggregate Functions | Returns the row, which is the specified number backward from the current row. Default is 1 if offset is omitted. | | [LAG\_VECTORS](/data-preparation#lag_vectors) | Data Preparation | Groups lagged columns generated by the `MULTI_LAGS` or `MULTI_LAGS_ZEROFILL` functions into vector columns. | | [LAGS](/data-preparation#lags) | Data Preparation | Generates a series of lagged columns for a single variable in one statement. | | [LAGS\_ZEROFILL](/data-preparation#lags_zerofill) | Data Preparation | Generates lagged columns for a single variable and replaces NULL values with `0`. | | [LAST\_VALUE](/window-aggregate-functions#last_value) | Window Aggregate Functions | Returns the last value in the ordered result set. | | [LCASE](/character-and-binary-functions#lcase) | Character and Binary Functions | Alias for LOWER. | | [LEAD](/window-aggregate-functions#lead) | Window Aggregate Functions | Returns the row, which is the specified number forward from the current row. Default is 1 if offset is omitted. | | [LEAKYRELU](/math-functions-and-operators#leakyrelu) | Math Functions | Returns the leaky rectified linear unit function of x. | | [LEAST](/other-functions-and-expressions#least) | Conditional Functions | Returns the smallest non-NULL value of all arguments, or NULL if all arguments are NULL. | | [LEFT](/character-and-binary-functions#left) | Character and Binary Functions | Return the number of characters in the string equal to the value integer. If the integer is negative, the function returns all but the last integer characters. | | [LEFT\_SHIFT](/math-functions-and-operators#left_shift) | Math Functions | Returns x shifted to the left by y bits. | | [LENGTH](/character-and-binary-functions#length) | Character and Binary Functions | For character data types, this value is in terms of characters. For binary data types, this value is in terms of bytes. | | [LN](/math-functions-and-operators#ln) | Math Functions | Returns the natural logarithm of x. | | [LOCATE](/character-and-binary-functions#locate) | Character and Binary Functions | Alias for POSITION. Returns the index position of the first occurrence of the character value substring in the character value string. | | [LOG](/math-functions-and-operators#log) | Math Functions | Returns the base 10 logarithm of x. The optional base argument specifies the numeral system to use. If unspecified, the function defaults to base 10. | | [LOG\_GAMMA](/math-functions-and-operators#log_gamma) | Math Functions | The natural logarithm of the absolute value of the gamma function. | | [LOG\_LOSS](/array-functions-and-operators) | Array Functions | Returns the log loss of two arrays. | | [LOG\_LOSS](/matrix-functions-and-operators) | Matrix Functions | Returns the log loss of two one-dimensional matrices or vectors. | | [LOG2](/math-functions-and-operators#log2) | Math Functions | Returns the base 2 logarithm of x. | | [LOGITS\_LOSS](/array-functions-and-operators) | Array Functions | Returns the logits loss of two arrays. | | [LOGITS\_LOSS](/matrix-functions-and-operators) | Matrix Functions | Returns the logits loss of two one-dimensional matrices or vectors. | | [LOOKUP](/load-data-from-external-sources-in-data-pipelines) | Special Data Pipeline Transformation Functions | Look up and load data in an external data source. | | [LOWER](/character-and-binary-functions#lower) | Character and Binary Functions | Alias for LCASE. Convert string to lowercase. | | [LPAD](/character-and-binary-functions#lpad) | Character and Binary Functions | Pad the input text to the specified length with the pad string on the left side. | | [LTRIM](/character-and-binary-functions#ltrim) | Character and Binary Functions | Removes leading blanks from the string value string. | | [LUPQ\_DECOMP](/matrix-functions-and-operators) | Matrix Functions | Returns LUPQ decomposition of a square matrix A as a tuple of 4 matrices where PAQ = LU. | | [MAP\_KEYS](/transform-data-in-data-pipelines#other-data-transformation-functions) | Special Data Pipeline Transformation Functions | Returns the keys in the specified JSON string. | | [MAP\_VALUES](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Returns the values in the specified JSON string. | | [MATRIX\_DIM](/matrix-functions-and-operators) | Matrix Functions | Returns the dimensions of the specified matrix as a tuple of (row, col) integers. | | [MAKE\_MATRIX\_IXJ](/matrix-functions-and-operators) | Matrix Functions | Creates an `ixj` matrix with elements `e_00`, `…`, `e_ij`. | | [MAKEDATETIME](/date-and-time-functions#makedatetime) | Date and Time Functions | Returns a timestamp consisting of the specified date and time. | | [MATRIX\_FROM\_TEXT](/matrix-functions-and-operators) | Matrix Functions | Creates a matrix from the specified string. | | [MATRIX\_TRACE](/matrix-functions-and-operators) | Matrix Functions | Returns trace of a square matrix as a double. | | [MAX](/aggregate-functions#max) | Aggregate Functions | Maximum value in the specified column. | | [MD5](/character-and-binary-functions#md5) | Character and Binary Functions | Returns the hexadecimal string (all lowercase) representing the md5 hash of char. | | [MEAN\_ABSOLUTE\_ERROR](/aggregate-functions#mean_absolute_error) | Aggregate Functions | Returns the mean absolute error (MAE) between actual and predicted values. | | [MEAN\_ABSOLUTE\_PERCENTAGE\_ERROR](/aggregate-functions#mean_absolute_percentage_error) | Aggregate Functions | Returns the mean absolute percentage error (MAPE) between actual and predicted values. | | [METADATA](/load-metadata-and-file-based-partitioned-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Extracts the metadata value for the specified key from available metadata for the pipeline. | | [MICROSECONDS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type microseconds. | | [MID](/character-and-binary-functions#mid) | Character and Binary Functions | Alias for SUBSTRING. | | [MILLISECOND](/date-and-time-functions#millisecond) | Date and Time Functions | Extracts the millisecond portion of a timestamp as an integer. | | [MILLISECONDS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type milliseconds. | | [MIN](/aggregate-functions#min) | Aggregate Functions | Minimum value in the specified column. | | [MINUTE](/date-and-time-functions#minute) | Date and Time Functions | Extracts the minute portion of a timestamp or date as an integer. | | [MINUTES](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type minutes. | | [MOD](/math-functions-and-operators#mod) | Math Functions | Returns the remainder from x divided by y. | | [MONTH](/date-and-time-functions#month) | Date and Time Functions | Extracts the month portion of a timestamp or date as an integer. | | [MONTH\_NAME](/date-and-time-functions#month_name) | Date and Time Functions | Returns the calendar name in English of the month for the specified date. | | [MONTHS](/scalar-data-conversion-functions)
| Scalar Data Conversion Functions | Converts an integral value to an interval value of type months to be used in date calculations. | | [MONTHS\_BETWEEN](/date-and-time-functions#months_between) | Date and Time Functions | Returns the difference between the two dates or timestamps in months as a DOUBLE. | | [MSECS](/date-and-time-functions#msecs) | Date and Time Functions | The seconds field, including fractional parts. The function multiplies the seconds part of the value by 1,000. | | [MULTI\_LAGS](/data-preparation#multi_lags) | Data Preparation | Generates lagged columns for multiple variables at once. | | [MULTI\_LAGS\_ZEROFILL](/data-preparation#multi_lags_zerofill) | Data Preparation | Generates lagged columns for multiple variables and replaces NULL values with `0`. | | [MURMUR3](/other-functions-and-expressions#murmur3) | Conditional Functions | [Returns a 32-bit MurmurHash3 hash of the input value as an INTEGER data type.](https://github.com/aappleby/smhasher/blob/master/README.md) | | [NANOS\_TO\_TIMESTAMP](/date-and-time-functions#nanos_to_timestamp) | Date and Time Functions | Convert a number of nanoseconds into a timestamp equivalent to the duration after the epoch time. | | [NANOSECONDS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type nanoseconds. | | [NEXT\_DAY](/date-and-time-functions#next_day) | Date and Time Functions | Returns the closest date after a specified date that lies on a specific day of the week. | | [NOW](/date-and-time-functions#now) | Date and Time Functions | Alias for CURRENT\_TIMESTAMP. | | [NTH\_VALUE](/window-aggregate-functions#nth_value) | Window Aggregate Functions | Returns the nth value in the ordered result set. | | [NULL\_IF](/other-functions-and-expressions#null_if) | Conditional Functions | Returns the NULL value if two arguments are equal; otherwise, returns the first argument. | | [NULL\_MATRIX](/matrix-functions-and-operators) | Matrix Functions | Returns a null matrix of the given (row, col) dimensions. | | [OCTET\_LENGTH](/character-and-binary-functions#octet_length) | Character and Binary Functions | Returns the length in bytes of a character or binary value. | | [PARSE\_DELIMITED\_ARRAY](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Converts a string of text data representing an array into a `CHAR[]`. | | [PERCENT\_RANK](/window-aggregate-functions#percent_rank) | Window Aggregate Functions | The value returned is 0 \< n ≤ 1 and can be used to calculate the percentage of values less than the current group, excluding the highest value. | | [PERCENTILE](/window-aggregate-functions#percentile) | Window Aggregate Functions | Returns the value that corresponds to the specified percentile (0 ≤ n ≤ 1) within the group. | | [PI](/math-functions-and-operators#pi) | Math Functions | Returns the constant value of π. | | [PMOD](/math-functions-and-operators#pmod) | Math Functions | Returns the smallest non-negative equivalence class of x % y. | | [POSITION](/character-and-binary-functions#position) | Character and Binary Functions | Alias for LOCATE. | | [POWER](/math-functions-and-operators#power) | Math Functions | Returns x raised to the power of y. | | [PRECISION\_SCORE](/aggregate-functions#precision_score) | Aggregate Functions | Returns the precision score for a specified positive class. | | [PROBIT](/math-functions-and-operators#probit) | Math Functions | The inverse of the cumulative distribution function. | | [PRODUCT](/aggregate-functions#product) | Aggregate Functions | Product over the set. | | [QR\_DECOMP](/matrix-functions-and-operators) | Matrix Functions | Returns QR decomposition of a matrix as a tuple of 2 matrices. | | [QUARTER](/date-and-time-functions#quarter) | Date and Time Functions | Returns an integer between 1 and 4 that represents the quarter of the year in which the specified date falls. | | [RADIANS](/math-functions-and-operators#radians) | Math Functions | Returns the corresponding angle in radians for x in degrees. | | [RAND](/math-functions-and-operators#rand) | Math Functions | Takes no argument and returns a random DOUBLE value in the range \[0, 1). | | [RAND\_UUID](/other-functions-and-expressions#rand_uuid)
| Other Functions and Expressions | Generates a random UUID value (version 4) | | [RANK](/window-aggregate-functions#rank) | Window Aggregate Functions | Assigns a number to each row in the result set with equal values having the same number. There can be gaps between ranks. | | [RATIO\_TO\_REPORT](/window-aggregate-functions#ratio_to_report) | Window Aggregate Functions | Computes the ratio of a value to the sum of the set of values. | | [RECALL\_SCORE](/aggregate-functions#recall_score) | Aggregate Functions | Returns the recall score for a specified positive class. | | [ROC\_AUC\_SCORE](/aggregate-functions#roc_auc_score) | Aggregate Functions | Returns the area under the receiver operating characteristic (ROC) curve (AUC). | | [RECORD\_UUID](/lat-transformation-functions#record_uuid)
| Other Transformation Functions | Returns string that represents a unique identifier for the record in a specified pipeline for a specified file\_group or topic. | | [REDUCE](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Applies a merge function to a starting value and all elements in the array, and then reduces the array to a single value. Optionally, specify a finish function for the returned single value. | | [REGEXP\_COUNT](/character-and-binary-functions#regexp_count) | Character and Binary Functions | Searches a string for all occurrences of a regular expression pattern. | | [REGEXP\_INSTR](/character-and-binary-functions#regexp_instr) | Character and Binary Functions | Searches a string using a regular expression pattern and returns an integer representing the start position or end position of the substring that matches. | | [REGEXP\_REPLACE](/character-and-binary-functions#regexp_replace) | Character and Binary Functions | Searches a string for all occurrences of a regular expression pattern. | | [REGEXP\_SUBSTR](/character-and-binary-functions#regexp_substr) | Character and Binary Functions | Returns one substring from a string that matches a specified regular expression pattern. | | [RELU](/math-functions-and-operators#relu) | Math Functions | Returns the rectified linear unit function of x. | | [REPEAT](/character-and-binary-functions#repeat) | Character and Binary Functions | Repeats the character value char a number of times equal to num. | | [REPLACE](/character-and-binary-functions#replace)

| Character and Binary Functions | Replaces all occurrences of `substr_to_remove` in the character value string with `substr_to_replace`. | | [REVERSE](/character-and-binary-functions#reverse) | Character and Binary Functions | Reverse the input string. | | [RIGHT](/character-and-binary-functions#right) | Character and Binary Functions | Return the number of trailing characters in the string equal to the value integer. | | [RIGHT\_SHIFT](/math-functions-and-operators#right_shift) | Math Functions | Returns x shifted to the right by y bits. | | [ROUND](/date-and-time-functions#round) | Date and Time Functions | Returns the specified date or timestamp, rounded to the specified precision. | | [ROUND](/math-functions-and-operators#round) | Math Functions | Returns x rounded to the nearest integer. | | [ROW\_NUMBER](/window-aggregate-functions#row_number) | Window Aggregate Functions | Assigns a unique number to each row in the result set. | | [RPAD](/character-and-binary-functions#rpad) | Character and Binary Functions | Pad the input text to the specified length with the pad string on the right side. | | [RSUBSTRING](/character-and-binary-functions#rsubstring) | Character and Binary Functions | Returns the substring from the right side of a string, based on a specified length. | | [RTRIM](/character-and-binary-functions#rtrim) | Character and Binary Functions | Removes leading blanks from the string value string. | | [SECOND](/date-and-time-functions#second) | Date and Time Functions | Extracts the seconds portion of a timestamp as an integer. | | [SECONDS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type seconds. | | [SHA1](/character-and-binary-functions#sha1) | Character and Binary Functions | Uses the \[SHA-1]\([https://en.wikipedia.org/wiki/SHA-1#:\~:text=In%20cryptography%2C%20SHA%2D1%20(,rendered%20as%2040%20hexadecimal%20digits](https://en.wikipedia.org/wiki/SHA-1#:~:text=In%20cryptography%2C%20SHA%2D1%20\(,rendered%20as%2040%20hexadecimal%20digits).) cryptographic hash function to convert a string into a 40-character string representing the hexadecimal value of a 160-bit checksum. | | [SHOW](/other-functions-and-expressions#show) | System Functions | The SHOW function enables you to explore the database and its metadata for user-defined items. | | [SIGN](/math-functions-and-operators#sign) | Math Functions | Returns the positive (+1), zero (0), or negative (-1) sign of x. | | [SIN](/math-functions-and-operators#sin) | Math Functions | Returns the sine of x. | | [SINH](/math-functions-and-operators#sinh) | Math Functions | Returns the hyperbolic sine of x. | | [SKEW](/aggregate-functions#skew) | Aggregate Functions | The sample over the set. | | [SKEWP](/aggregate-functions#skewp) | Aggregate Functions | Computes the population skewness over the set of values. | | [SMALLINT](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type `smallint`. | | [SOFTMAX](/array-functions-and-operators) | Array Functions | Returns the softmax of the array. | | [SOFTMAX](/matrix-functions-and-operators) | Matrix Functions | Returns the softmax of a one-dimensional matrix or vector. | | [SPACE](/character-and-binary-functions#space) | Character and Binary Functions | Returns a string of repeated spaces equal to the number value, repeat. | | [SPLIT\_PART](/character-and-binary-functions#split_part) | Character and Binary Functions | Split the value string based on the delimiter value. The function returns a substring from the split operation based on the index value (starting from 1). | | [SPLIT\_TO\_ARRAY](/character-and-binary-functions#split_to_array) | Character and Binary Functions | Splits a string into an array of substrings. | | [SQRT](/math-functions-and-operators#sqrt) | Math Functions | Returns the square root of x. | | [SQUARE](/math-functions-and-operators#square) | Math Functions | Returns the square of x. | | [ST\_ADDPOINT](/linestring-functions#st_addpoint) | Geospatial Linestring Function | Adds a POINT to the given LINESTRING at the specified 0-indexed location. | | [ST\_ANGLE](/spatial-measurement#st_angle) | Geospatial Spatial Measurement | Calculates the angle between two lines. | | [ST\_AREA](/spatial-measurement#st_area) | Geospatial Spatial Measurement | Returns the area of the specified geospatial object in the specified unit of measurement. | | [ST\_ASBINARY](/conversion-functions#st_asbinary) | Geospatial Conversion Functions | Returns the well-known binary (WKB) representation of the specified geography. Alias of ST\_ASWKB. | | [ST\_ASEWKT](/conversion-functions#st_asewkt) | Geospatial Conversion Functions | Returns a string that represents geographic coordinates of a specified POINT in the specified format. | | [ST\_ASGEOJSON](/conversion-functions#st_asgeojson) | Geospatial Conversion Functions | Alias of ST\_ASBINARY. | | [ST\_ASLATLONTEXT](/conversion-functions#st_aslatlontext) | Geospatial Conversion Functions | [Returns the GeoJSON representation of the specified geography using the IETF standards.](https://datatracker.ietf.org/doc/html/rfc7946) | | [ST\_ASTEXT](/conversion-functions#st_astext) | Geospatial Conversion Functions | Alias of ST\_ASTEXT. | | [ST\_ASWKB](/conversion-functions#st_aswkb) | Geospatial Conversion Functions | Alias of ST\_ASTEXT. | | [ST\_ASWKT](/conversion-functions#st_aswkt) | Geospatial Conversion Functions | Alias of ST\_ASWKT and ST\_EWKT. Returns the WKT representation of the specified geography. | | [ST\_AZIMUTH](/spatial-measurement#st_azimuth) | Geospatial Spatial Measurement | Returns the azimuth of the line from `point1` to `point2` in radians. | | [ST\_BOUNDINGDIAGONAL](/spatial-operators#st_boundingdiagonal) | Geospatial Spatial Operators | Returns the diagonal LINESTRING from the minimum point to the maximum point of the bounding box that ST\_ENVELOPE returns. | | [ST\_BUFFER](/spatial-operators#st_buffer) | Geospatial Spatial Operators | Returns a geography that contains all points where the distance from the geography is less than or equal to the specified distance. | | [ST\_CENTROID](/point-constructors#st_centroid) | Geospatial Point Constructors | The geographic center of mass is calculated by taking the average of all points on a three-dimensional sphere, projecting the resultant point onto the sphere, and converting it back to latitude and longitude coordinates. | | [ST\_CLOSESTPOINT](/spatial-operators#st_closestpoint) | Geospatial Spatial Operators | Returns the two-dimensional POINT of one specified geospatial object that is closest to a second specified geospatial object. | | [ST\_CLUSTERDBSCAN](/spatial-relationships#st_clusterdbscan) | Geospatial Spatial Relationships | [Returns the cluster number for each input geography, based on a two-dimensional implementation of the density-based spatial clustering of applications with noise (DBSCAN) algorithm.](https://en.wikipedia.org/wiki/DBSCAN) | | [ST\_CONTAINS](/spatial-relationships#st_contains) | Geospatial Spatial Relationships | Returns TRUE if the first geographic argument, geo1, contains the second geographic argument, geo2. | | [ST\_CONTAINSPROPERLY](/spatial-relationships#st_containsproperly) | Geospatial Spatial Relationships | Returns true if geo2 lies entirely in the interior of geo1, and does not intersect or touch the boundary or exterior points. | | [ST\_CONVEXHULL](/spatial-operators#st_convexhull) | Geospatial Spatial Operators | The convex hull is the smallest convex geometry that encloses the input geometry. | | [ST\_COORDDIM](/attribute-functions#st_coorddim) | Geospatial Attribute Functions | Alias for ST\_NDIMS or ST\_NDIMENSION. Returns an INTEGER of the coordinate dimension of the specified geography. | | [ST\_COVEREDBY](/spatial-relationships#st_coveredby) | Geospatial Spatial Relationships | Returns TRUE if no POINT in geo1 is outside of geo2. | | [ST\_COVERS](/spatial-relationships#st_covers) | Geospatial Spatial Relationships | Returns TRUE if no POINT in geo2 is outside of geo1. | | [ST\_CROSSES](/spatial-relationships#st_crosses) | Geospatial Spatial Relationships | Returns TRUE if two geospatial objects meet these criteria: The intersection of the geospatial interiors is not empty. The intersection is not equal to geo1 or geo2. Neither geospatial object is a single POINT. | | [ST\_DIFFERENCEARRAY](/spatial-operators#st_differencearray) | Geospatial Spatial Operators | Returns an array containing any geospatial objects that are present in the first specified geospatial argument that are not found in the second geospatial argument. | | [ST\_DIMENSION](/attribute-functions#st_dimension) | Geospatial Attribute Functions | Returns an INTEGER that represents the dimension of the specified geography. | | [ST\_DISJOINT](/spatial-relationships#st_disjoint) | Geospatial Spatial Relationships | Returns true if the specified geographies have no intersection, including boundaries. Both geographic arguments can be different types. | | [ST\_DISTANCE](/spatial-measurement#st_distance)
| Geospatial Spatial Measurement | Returns the minimum distance between the specified arguments. | | [ST\_DISTANCE](/spatiotemporal-measurement#st_distance) | Geospatial Spatiotemporal Measurement | Returns the two-dimensional interpolated minimum simultaneous distance between two LINESTRING-TIMESTAMP array pairs in the specified unit of measurement. | | [ST\_DISTANCESPHERE](/spatial-measurement#st_distancesphere) | Geospatial Spatial Measurement | Returns the minimum distance between the specified arguments using a spherical computation. | | [ST\_DISTANCESPHEROID](/spatial-measurement#st_distancespheroid) | Geospatial Spatial Measurement | Returns the minimum distance between the specified arguments using a spheroid computation. | | [ST\_DWITHIN](/spatial-relationships#st_dwithin) | Geospatial Spatial Relationships | Returns TRUE if the geographies are within a specified distance in meters. | | [ST\_ENDPOINT](/linestring-functions#st_endpoint) | Geospatial Linestring Function | Returns the endpoint of a specified LINESTRING. The returned value is a POINT. | | [ST\_ENVELOPE](/spatial-operators#st_envelope) | Geospatial Spatial Operators | Returns a POLYGON that represents the minimum bounding box for the specified geography. | | [ST\_EQUALS](/spatial-relationships#st_equals) | Geospatial Spatial Relationships | Returns TRUE if both geographies are spatially equal. | | [ST\_EUCLIDEANDISTANCE3D](/spatial-measurement#st_euclideandistance3d) | Geospatial Spatial Measurement | Returns the minimum distance between two geospatial points with altitude values. | | [ST\_EXPAND](/spatial-operators#st_expand) | Geospatial Spatial Operators | Returns the bounding box of a specified geospatial value, which is expanded by a specified length. | | [ST\_EXTERIORRING](/spatial-operators#st_exteriorring) | Geospatial Spatial Operators | Returns a LINESTRING that represents the exterior ring of a provided POLYGON value. | | [ST\_FLIPCOORDINATES](/spatial-operators#st_flipcoordinates) | Geospatial Spatial Operators | Returns a new geographic object with the X and Y coordinates switched using the specified argument. | | [ST\_FORCE2D](/spatial-operators#st_force2d) | Geospatial Spatial Operators | Convert a geographic object into a two-dimensional geography. | | [ST\_FORCECCW](/polygon-constructors#st_forceccw) | Geospatial Polygon Constructors | Creates a standardized polygon from an existing one. defines standardized as the exterior being counterclockwise (CCW) and all holes being clockwise (CW) oriented. | | [ST\_GEOGPOINT](/point-constructors#st_geogpoint) | Geospatial Point Constructors | Creates and returns a POLYGON geography with a single point, defined by the longitude and latitude specified for the function. | | [ST\_GEOHASH](/conversion-functions#st_geohash) | Geospatial Conversion Functions | Returns a string that represents the geohash of the input POINT. | | [ST\_GEOMETRYTYPE](/attribute-functions#st_geometrytype) | Geospatial Attribute Functions | Returns a string representing the geometry type of the input value. | | [ST\_HAUSDORFFDISTANCE](/spatial-measurement#st_hausdorffdistance) | Geospatial Spatial Measurement | Returns the Hausdorff distance between two geographies in a specified measurement. | | [ST\_INTERIORRINGN](/spatial-operators#st_interiorringn) | Geospatial Spatial Operators | Returns a LINESTRING representing the interior ring of the specified POLYGON, which is specified by its index. (1-indexed) | | [ST\_INTERSECTALL](/spatial-operators#st_intersectall) | Geospatial Spatial Operators | Returns the intersection of all geographies in the specified array. All geographies in the array must be the same type. | | [ST\_INTERSECTION](/spatiotemporal-operators#st_intersection) | Geospatial Spatiotemporal Operators | Returns a tuple that represents the intersection of a spatiotemporal LINESTRING with a static geography. | | [ST\_INTERSECTIONARRAY](/spatial-operators#st_intersectionarray) | Geospatial Spatial Operators | Returns a geography that represents the point-set intersection of two geographies. | | [ST\_INTERSECTS](/spatial-relationships#st_intersects) | Geospatial Spatial Relationships | Returns TRUE if the specified geographies have an intersection, including boundaries. | | [ST\_ISCCW](/spatial-relationships#st_isccw) | Geospatial Spatial Relationships | Alias for ST\_ISPOLYGONCCW. | | [ST\_ISCLOSED](/spatial-relationships#st_isclosed) | Geospatial Spatial Relationships | Returns TRUE if an input POLYGON has an exterior that is counter-clockwise. | | [ST\_ISEMPTY](/attribute-functions#st_isempty) | Geospatial Attribute Functions | Returns TRUE if the specified geography value is empty, such as 'POLYGON EMPTY'. | | [ST\_ISPOLYGONCCW](/spatial-relationships#st_ispolygonccw) | Geospatial Spatial Relationships | Returns TRUE if an input LINESTRING has starting and ending points that are equal. | | [ST\_ISPOLYGONCW](/spatial-relationships#st_ispolygoncw) | Geospatial Spatial Relationships | Returns TRUE if an input POLYGON has an exterior that is clockwise. | | [ST\_ISRING](/spatial-relationships#st_isring) | Geospatial Spatial Relationships | Returns TRUE if the specified LINESTRING is closed and does not intersect itself. | | [ST\_ISSIMPLE](/spatial-relationships#st_issimple) | Geospatial Spatial Relationships | For LINESTRING values, this function returns FALSE if any line segments intersect anywhere besides the endpoints. For POLYGON values, the function returns FALSE if either the exterior ring or any interior hole is not simple. | | [ST\_ISVALID](/spatial-relationships#st_isvalid) | Geospatial Spatial Relationships | [Returns TRUE if the specified geospatial value is a well-formed and valid geography according to the OGC standards.](https://www.ogc.org/standard/sfa/) | | [ST\_LENGTH](/spatial-measurement#st_length) | Geospatial Spatial Measurement | Returns the length of the specified line in the specified measurement unit. | | [ST\_LENGTH2D](/spatial-measurement#st_length2d) | Geospatial Spatial Measurement | Alias for ST\_LENGTH. | | [ST\_LINEFROMEWKT](/linestring-constructors#st_linefromewkt) | Geospatial Linestring Constructor | Creates a LINESTRING from the specified CHAR. | | [ST\_LINEFROMGEOJSON](/linestring-constructors#st_linefromgeojson) | Geospatial Linestring Constructor | Creates a LINESTRING represented by the specified GeoJSON. | | [ST\_LINEFROMTEXT](/linestring-constructors#st_linefromtext) | Geospatial Linestring Constructor | Creates a LINESTRING from a specified CHAR. The CHAR must be a LINESTRING value in WKT format. | | [ST\_LINEFROMWKB](/linestring-constructors#st_linefromwkb) | Geospatial Linestring Constructor | Creates a LINESTRING from the specified BINARY. The BINARY value must be a LINESTRING in WKB format. | | [ST\_LINEGETALLTIMESATPOINT](/spatiotemporal-operators#st_linegetalltimesatpoint) | Geospatial Spatiotemporal Operators | Returns a timestamp array of all times when the specified LINESTRING value intersects the specified POINT value. | | [ST\_LINEGETPOINTATTIME](/spatiotemporal-operators#st_linegetpointattime) | Geospatial Spatiotemporal Operators | Returns a POINT within the bounds of the specified LINESTRING that corresponds to the interpolated point at the specified TIMESTAMP value. | | [ST\_LINEGETTIMEATPOINT](/spatiotemporal-operators#st_linegettimeatpoint) | Geospatial Spatiotemporal Operators | Returns the interpolated time of the specified POINT on the specified LINESTRING that is paired with a TIMESTAMP ARRAY. | | [ST\_LINEINTERPOLATEPOINT](/linestring-functions#st_lineinterpolatepoint) | Geospatial Linestring Function | Returns a POINT along a LINESTRING based on a specified fraction of its total length. | | [ST\_LINELOCATEPOINT](/linestring-functions#st_linelocatepoint) | Geospatial Linestring Function | Similar to ST\_LINEINTERPOLATEPOINT, this function computes a fraction based on where a specified POINT is located along the length of a specified LINESTRING. | | [ST\_LINESTRING](/linestring-constructors#st_linestring) | Geospatial Linestring Constructor | Creates a LINESTRING based on the specified inputs. | | [ST\_LINESUBSTRING](/linestring-functions#st_linesubstring) | Geospatial Linestring Function | Returns a LINESTRING that is a substring of a specified line that starts and ends at the specified fractions of its total length. | | [ST\_LONGESTLINE](/spatial-operators#st_longestline) | Geospatial Spatial Operators | Returns the longest LINESTRING between two given geospatial arguments. | | [ST\_LONGESTLINE](/spatiotemporal-operators#st_longestline) | Geospatial Spatiotemporal Operators | With the specified two LINESTRING-TIMESTAMP ARRAY pairs, this function returns a two-point LINESTRING that represents the maximum distance between points at a concurrent time. | | [ST\_MAKEENVELOPE](/spatial-operators#st_makeenvelope) | Geospatial Spatial Operators | Returns a POLYGON with vertices that represent the minimum bounding box for the specified coordinates. | | [ST\_MAKELINE](/linestring-constructors#st_makeline) | Geospatial Linestring Constructor | Alias for ST\_LINESTRING. | | [ST\_MAKEPOINT](/point-constructors#st_makepoint) | Geospatial Point Constructors | Alias for ST\_POINT. | | [ST\_MAKEPOLYGON](/polygon-constructors#st_makepolygon) | Geospatial Polygon Constructors | Alias for ST\_POLYGON. | | [ST\_MAXDISTANCE](/spatial-measurement#st_maxdistance) | Geospatial Spatial Measurement | Returns maximum distance between the specified arguments. | | [ST\_MAXDISTANCE](/spatiotemporal-measurement#st_maxdistance) | Geospatial Spatiotemporal Measurement | Returns the two-dimensional interpolated maximum cotemporal distance between two LINESTRING-TIMESTAMP array pairs in the specified unit of measurement. | | [ST\_MEMSIZE](/attribute-functions#st_memsize) | Geospatial Attribute Functions | Returns an INTEGER representing the number of bytes in memory required to store the specified geography. | | [ST\_MINIMUMBOUNDINGCIRCLE](/spatial-operators#st_minimumboundingcircle) | Geospatial Spatial Operators | Returns the smallest circle POLYGON that contains the specified geographic object. | | [ST\_MULTIDIFFERENCEARRAY](/spatial-operators#st_multidifferencearray) | Geospatial Spatial Operators | Returns an array of geographies that represents the parts of the union of the geographies in the first array that do not intersect with the union of geographies in the second array. | | [ST\_MINIMUMDISTANCETOSURFACE](/spatial-measurement#st_minimumdistancetosurface) | Geospatial Spatial Measurement | Calculates the shortest distance between any point along a Euclidean line segment in three-dimensional space and the surface of the Earth. | | [ST\_MULTIINTERSECTIONARRAY](/spatial-operators#st_multiintersectionarray) | Geospatial Spatial Operators | When you specify two arrays of geospatial objects, this function returns an array of any intersections. | | [ST\_MULTISYMDIFFERENCEARRAY](/spatial-operators#st_multisymdifferencearray) | Geospatial Spatial Operators | When you specify two arrays of geospatial objects, this function returns an array of any geospatial values that do not intersect. | | [ST\_MULTIUNIONARRAY](/spatial-operators#st_multiunionarray) | Geospatial Spatial Operators | When you specify two arrays of geospatial objects, this function returns one array that represents a union of all geographies in both arrays. | | [ST\_NDIMENSION](/attribute-functions#st_ndims-or-st_ndimension) | Geospatial Attribute Functions | Alias for ST\_COORDDIM. | | [ST\_NDIMS](/attribute-functions) | Geospatial Attribute Functions | Alias for ST\_COORDDIM. | | [ST\_NPOINTS](/attribute-functions#st_npoints-or-st_numpoints) | Geospatial Attribute Functions | Returns an INTEGER representing the number of POINT values in a specified geography. | | [ST\_NRINGS](/spatial-operators#st_nrings) | Geospatial Spatial Operators | Returns the number of rings of the specified POLYGON, including both interior and exterior rings. | | [ST\_NUMINTERIORRING](/spatial-operators#st_numinteriorrings-or-st_numinteriorring) | Geospatial Spatial Operators | Returns the number of interior rings of the specified POLYGON. | | [ST\_NUMINTERIORRINGS](/spatial-operators) | Geospatial Spatial Operators | Returns the number of interior rings of the specified POLYGON. | | [ST\_NUMPOINTS](/attribute-functions) | Geospatial Attribute Functions | Returns an INTEGER representing the number of POINT values in a specified geography. | | [ST\_OVERLAPS](/spatial-relationships#st_overlaps) | Geospatial Spatial Relationships | Returns TRUE if both geographic arguments are of the same dimension and they intersect each other, but neither contains the other. | | [ST\_PERIMETER](/spatial-measurement#st_perimeter) | Geospatial Spatial Measurement | Returns the length of the exterior (outer ring) of the POLYGON in the specified unit of measurement. | | [ST\_PERIMETER2D](/spatial-measurement#st_perimeter2d) | Geospatial Spatial Measurement | Alias for ST\_PERIMETER. | | [ST\_POINT](/point-constructors#st_point) | Geospatial Point Constructors | Creates a POINT from the specified input arguments. | | [ST\_POINTFROMEWKT](/point-constructors#st_pointfromewkt) | Geospatial Point Constructors | Alias for ST\_POINT. Creates a POINT using an EWKT-formatted CHAR as an input argument. | | [ST\_POINTFROMGEOHASH](/point-constructors#st_pointfromgeohash) | Geospatial Point Constructors | Creates a POINT represented by the specified geohash. | | [ST\_POINTFROMGEOJSON](/point-constructors#st_pointfromgeojson) | Geospatial Point Constructors | Creates a POINT represented by the specified GeoJSON value as an input argument. | | [ST\_POINTFROMTEXT](/point-constructors#st_pointfromtext) | Geospatial Point Constructors | Alias for ST\_POINT(char). | | [ST\_POINTFROMWKB](/point-constructors#st_pointfromwkb) | Geospatial Point Constructors | Alias for ST\_POINT(binary). | | [ST\_POINTINSIDECIRCLE](/spatial-relationships#st_pointinsidecircle) | Geospatial Spatial Relationships | Returns TRUE if the geographic object is inside a circle that is centered at the specified point coordinates and with the specified radius. | | [ST\_POINTN](/linestring-functions#st_pointn) | Geospatial Linestring Function | Returns the POINT value at a specified index of the given LINESTRING. | | [ST\_POINTONSURFACE](/spatial-operators#st_pointonsurface) | Geospatial Spatial Operators | Returns a POINT guaranteed to intersect the specified geospatial object. | | [ST\_POLYGON](/polygon-constructors#st_polygon) | Geospatial Polygon Constructors | Creates a POLYGON. | | [ST\_POLYGONFROMEWKT](/polygon-constructors#st_polygonfromewkt) | Geospatial Polygon Constructors | Creates a POLYGON using an EWKT-formatted CHAR as an input argument. Alias for the ST\_POLYGON constructor. | | [ST\_POLYGONFROMGEOJSON](/polygon-constructors#st_polygonfromgeojson) | Geospatial Polygon Constructors | Creates a POLYGON from the specified POINT, POINT array, LINESTRING, or POLYGON geography. | | [ST\_POLYGONFROMTEXT](/polygon-constructors#st_polygonfromtext) | Geospatial Polygon Constructors | Alias for ST\_POLYGON(char). | | [ST\_POLYGONFROMWKB](/polygon-constructors#st_polygonfromwkb) | Geospatial Polygon Constructors | Alias for ST\_POLYGON(binary). | | [ST\_PROJECT](/spatial-operators#st_project) | Geospatial Spatial Operators | Returns a POINT by projecting a distance and an azimuth value from the specified starting POINT value. | | [ST\_REDUCEPRECISION](/spatial-operators#st_reduceprecision) | Geospatial Spatial Operators | Returns a new geospatial object with all POINT values rounded to the specified decimal precision. | | [ST\_RELATE](/spatial-relationships#st_relate) | Geospatial Spatial Relationships | Returns the [DE-9IM](https://postgis.net/workshops/postgis-intro/de9im.html) intersection string that represents the nature of the intersection with the specified geographies. | | [ST\_REMOVEPOINT](/linestring-functions#st_removepoint) | Geospatial Linestring Functions | Removes a POINT value at a specified index from the specified line. | | [ST\_REMOVEREPEATEDPOINTS](/spatial-operators#st_removerepeatedpoints) | Geospatial Spatial Operators | Returns a new geospatial object with no repeated POINT values. | | [ST\_REVERSE](/spatial-operators#st_reverse) | Geospatial Spatial Operators | Returns a new geospatial object with the vertexes reversed. | | [ST\_SEGMENTIZE](/spatial-operators#st_segmentize) | Geospatial Spatial Operators | Returns a geospatial object that the function modifies to have no segment longer than the specified max\_segment\_length in meters. | | [ST\_SETPOINT](/linestring-functions#st_setpoint) | Geospatial Linestring Functions | Replaces a POINT value in a given LINESTRING at a specified index. The function returns the altered LINESTRING with the replaced point. | | [ST\_SHORTESTLINE](/spatial-operators#st_shortestline) | Geospatial Spatial Operators | Returns the shortest LINESTRING between two specified geospatial arguments. | | [ST\_SHORTESTLINE](/spatiotemporal-operators#st_shortestline) | Geospatial Spatiotemporal Operators | When you specify two LINESTRING-TIMESTAMP ARRAY pairs, this function returns a LINESTRING with two points that represents the minimum distance between points at a concurrent time. | | [ST\_SIMPLIFY](/spatial-operators#st_simplify) | Geospatial Spatial Operators | Returns a simplified version of the specified geography, which is either a POINT or LINESTRING. | | [ST\_SIMPLIFYARRAY](/spatial-operators#st_simplifyarray) | Geospatial Spatial Operators | Returns a POLYGON array that represents a simplified version of the specified geography, which is either a POINT, LINESTRING, or POLYGON. | | [ST\_SNAPTOGRID](/spatial-operators#st_snaptogrid) | Geospatial Spatial Operators | Returns a new geography value with all POINT values rounded to the specified precisions. | | [ST\_SRID](/attribute-functions#st_srid) | Geospatial Attribute Functions | Returns the EPSG code of the spatial reference identifier (SRID) of the input geography. | | [ST\_STARTPOINT](/linestring-functions#st_startpoint)
| Geospatial Linestring Functions | Returns the starting POINT value of the line. | | [ST\_SYMDIFFERENCEARRAY](/spatial-operators#st_symdifferencearray) | Geospatial Spatial Operators | Returns a geographic array that contains the parts that are not common between two geographic objects, geo1 and geo2. | | [ST\_TOTALSECONDSININTERSECTION](/spatiotemporal-measurement#st_totalsecondsinintersection) | Geospatial Spatiotemporal Measurement | Returns the total number of seconds spent in the intersection result calculated by the spatiotemporal version of ST\_INTERSECTION. | | [ST\_TOUCHES](/spatial-relationships#st_touches) | Geospatial Spatial Relationships | Returns TRUE if the only POINT values in common between the two geographic arguments lie in the union of their boundaries. | | [ST\_UNIONARRAY](/spatial-operators#st_unionarray) | Geospatial Spatial Operators | Performs a union of the input geography values to produce a geographic array. | | [ST\_WHOLEEARTH](/polygon-constructors#st_wholeearth) | Geospatial Polygon Constructors | Returns the database internal representation of the whole earth polygon. | | [ST\_WITHIN](/spatial-relationships#st_within) | Geospatial Spatial Relationships | Alias for ST\_CONTAINS. | | [ST\_X](/attribute-functions#st_x) | Geospatial Attribute Functions | Returns the x value of the specified POINT. | | [ST\_XMAX](/attribute-functions#st_xmax) | Geospatial Attribute Functions | Returns the maximum x value of the specified geography. | | [ST\_XMIN](/attribute-functions#st_xmin) | Geospatial Attribute Functions | Returns the minimum x value of the specified geography. | | [ST\_Y](/attribute-functions#st_y) | Geospatial Attribute Functions | Returns the y value of the specified POINT. | | [ST\_YMAX](/attribute-functions#st_ymax) | Geospatial Attribute Functions | Returns the maximum y value of specified geography. | | [ST\_YMIN](/attribute-functions#st_ymin) | Geospatial Attribute Functions | Returns the minimum y value of specified geography. | | [STARTSWITH](/character-and-binary-functions#startswith) | Character and Binary Functions | Returns true if string starts with substring and false otherwise. | | [STDEV](/aggregate-functions#stdev) | Aggregate Functions | Sample standard deviation. | | [STDDEV](/aggregate-functions#stddev) | Aggregate Functions | Alias for STDEV. | | [STDDEV\_POP](/aggregate-functions#stddev_pop) | Aggregate Functions | Alias for STDEVP. | | [STDDEV\_SAMP](/aggregate-functions#stddev_samp) | Aggregate Functions | Alias for STDEV. | | [STDEVP](/aggregate-functions#stdevp) | Aggregate Functions | Population standard deviation. | | [STRING\_AGG](/aggregate-functions#string_agg) | Aggregate Functions | Returns a string concatenated from every row from the expression. The delimiter argument is optional. | | [STRING\_TO\_ARRAY](/array-functions-and-operators) | Array Functions | Converts the string representation of an array (e.g., `'int[1,2,NULL]'`) into an array. | | [STRING\_TO\_TUPLE](/tuple-functions-and-operators) | Tuple Functions | Converts the string representation of a tuple (e.g `'tuple<>(1,2,NULL)'`) into a tuple. | | [STRPOS](/character-and-binary-functions#strpos) | Character and Binary Functions | Equivalent to using LOCATE as `LOCATE(substring, string)`. Note the reversed argument order. | | [SUBSTR](/character-and-binary-functions#substr) | Character and Binary Functions | Alias for SUBSTRING. | | [SUBSTRING](/character-and-binary-functions#substring) | Character and Binary Functions | Returns the substring of a character or binary value. | | [SUBNET](/network-type-functions#subnet) | Network Type Functions | Computes the prefix from an `IP` or `IPV4` value and the size of the prefix. | | [SUM](/aggregate-functions#sum) | Aggregate Functions | Sum over the set. | | [SVD\_DECOMP](/matrix-functions-and-operators) | Matrix Functions | Returns SVD decomposition of a matrix as a tuple of 3 matrices. | | [TAN](/math-functions-and-operators#tan) | Math Functions | Returns the tangent of x. | | [TANH](/math-functions-and-operators#tanh) | Math Functions | Returns the hyperbolic tangent of x. | | [TIME](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses the string to create a time value. The string must be in the form `'HH:MM[.SSSSSSSSS]'`. | | [TIMESTAMP](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses a string and makes a timestamp value. The string must be in the format `'YYYY-MM-DD[ HH:MM][.SSSSSSSSS]'`. | | [TIMESTAMP\_TO\_NANOS](/date-and-time-functions#timestamp_to_nanos) | Date and Time Functions | Convert timestamp into nanoseconds after epoch as BIGINT. | | [TO\_ARRAY\_LENGTH](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Create the specified number of copies of any JSON object in an array. | | [TO\_BASE](/math-functions-and-operators#to_base) | Math Functions | Converts an integer value to its string representation in a specified base (radix). | | [TO\_CHAR](/character-and-binary-functions#to_char) | Character and Binary Functions | Converts a numeric, date, or timestamp value into a CHAR date type. | | [TO\_DATE](/formatting-functions#to_date) | Formatting Functions | Converts a character value with the specified format to a DATE type. | | [TO\_NUMBER](/formatting-functions#to_number) | Formatting Functions | Converts a character value with the specified format to a DECIMAL type. | | [TO\_TIMESTAMP](/formatting-functions#to_timestamp) | Formatting Functions | Converts a character value with the specified format to a TIMESTAMP type. | | [TRANSFORM](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Transforms an array based on the logic in a lambda expression. | | [TRANSLATE](/character-and-binary-functions#translate) | Character and Binary Functions | Replaces specified characters in a provided string with a separate set of characters. | | [TRANSPOSE](/matrix-functions-and-operators) | Matrix Functions | Returns transpose of the matrix. | | [TRIM](/character-and-binary-functions#trim) | Character and Binary Functions | Alias for BTRIM. Trim leading and trailing blanks from the string. | | [TRUNC](/math-functions-and-operators#trunc) | Math Functions | Returns x truncated to y decimal places. | | [TRUNCATE](/math-functions-and-operators#truncate) | Math Functions | Alias for TRUNC. | | [TUPLE()](/tuple-functions-and-operators) | Tuple Functions | Construct a tuple with the specified elements. Types of the tuple are inferred from the inner elements. | | [TUPLE\<\<>>](/tuple-functions-and-operators) | Tuple Functions | Construct a tuple with the specified elements. NULL is also supported as an element. | | [TYPE\[\]](/array-functions-and-operators) | Array Functions | Construct an array of SQL type TYPE giving the elements. | | [TYPE\_STRIP](/other-functions-and-expressions#type_strip) | Other Functions and Expressions | Returns the SQL type of the specified value. | | [UCASE](/character-and-binary-functions#ucase) | Character and Binary Functions | Alias for UPPER. | | [UNNEST](/array-functions-and-operators#unnest) | Array Functions | Expand each element in an input array out to an individual row. | | [UPPER](/character-and-binary-functions#upper) | Character and Binary Functions | Convert string to upper case. | | [USECS](/date-and-time-functions#usecs) | Date and Time Functions | The seconds part of a time value, including fractional parts, returned as an integer. The function multiplies the seconds part of the value by 1,000,000. | | [UUID](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses the string and makes a Universally Unique IDentifier (UUID) value. The string must be a valid UUID. | | [UUID\_GENERATE](/other-functions-and-expressions#uuid_generate) | Other Functions and Expressions | Generates a random UUID value (version 4). | | [VAR\_POP](/aggregate-functions#var_pop) | Aggregate Functions | Alias for VARIANCEP. | | [VAR\_SAMP](/aggregate-functions#var_samp) | Aggregate Functions | Alias for VARIANCE. | | [VARIANCE](/aggregate-functions#variance) | Aggregate Functions | Sample variance. | | [VARIANCEP](/aggregate-functions#variancep) | Aggregate Functions | Population variance. | | [VECTOR\_ARGMAX](/matrix-functions-and-operators) | Matrix Functions | Returns the argmax of a one-dimensional matrix or vector. | | [VECTOR\_ARGMIN](/matrix-functions-and-operators) | Matrix Functions | Returns the argmin of a one-dimensional matrix or vector. | | [VECTOR\_MAX](/matrix-functions-and-operators) | Matrix Functions | Returns the maximum of elements in a one-dimensional matrix/vector. | | [VECTOR\_MIN](/matrix-functions-and-operators) | Matrix Functions | Returns the minimum of elements in a one-dimensional matrix/vector. | | [VECTOR\_SUM](/matrix-functions-and-operators) | Matrix Functions | Returns the sum of elements in a one-dimensional matrix/vector. | | [VERSION](/other-functions-and-expressions#version) | System Functions | Returns the version of the database to which the client is currently connected. | | [WEEK](/date-and-time-functions#week) | Date and Time Functions | Returns the ISO-8601 week number, as an integer, of the specified timestamp or date value. | | [WEEKS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type weeks to be used in date calculations. | | [WIDTH\_BUCKET](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Creates `N` equal-width buckets in the range `[min,max)` as a histogram. | | [YEAR](/date-and-time-functions#year) | Date and Time Functions | Extracts the year portion of a timestamp or date as an integer. | | [YEARS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type years. | | [ZERO\_MATRIX](/matrix-functions-and-operators) | Matrix Functions | Returns a zero matrix of the specified (row, col) dimensions. | | [ZIP\_WITH](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Merges two or more input arrays, element-wise, into a single array using the trailing combining function. | | [ZN](/other-functions-and-expressions#zn) | Conditional Functions | If x is NULL, returns 0. Otherwise, returns x. | | [ZSCORE](/window-aggregate-functions#zscore) | Window Aggregate Functions | Zscore of the sample based on the `stddev()` function. | | [ZSCOREP](/window-aggregate-functions#zscorep) | Window Aggregate Functions | Zscore of the sample based on the `stddevp()` function. | # Alter Default Data Pipeline Behavior Source: https://docs.ocient.com/alter-default-data-pipeline-behavior Use the ALTER SYSTEM ALTER CONFIG SQL command in Ocient to tune data pipeline parameters such as execution time, partition counts, and type inference. An administrator might configure some parameters that control the behavior of data pipelines to modify the default behavior across the System or specific nodes. You can use the `ALTER SYSTEM ALTER CONFIG` SQL statements to control the configuration parameters. **Syntax** ```sql SQL theme={null} ALTER SYSTEM ALTER CONFIG SET 'sql.pipelineparameters.parameter_name' 'parameter_value'; ``` | **Parameter** | **Data** **Type** | **Description** | | ----------------- | ----------------- | ------------------------------------------------------------------------------------ | | `parameter_name` | string | The name of the pipeline parameter, either general or specific to the BINARY format. | | `parameter_value` | string | The value for the pipeline parameter. | **General Parameters** | **General Parameter Name** | **Default** | **Data Type** | **Required Value and Description** | | ---------------------------------------------------------- | ----------- | ---------------- | ------------------------------------------------------------------------------------------------------------------- | | `sql.pipelineparameters.targetTimePerExtractorTask` | 600 | `UINT64_T`
| Target execution time in seconds for `run_extractor` tasks for file loads. | | `sql.pipelineparameters.defaultPartitionsPerExtractorTask` | -1 | `INT` | Number of partitions per task.
If you set this parameter to -1, the number is based on available CPU cores. | | `sql.pipelineparameters.defaultExtractorCores` | -1 | `INT` | Number of cores to use in a task.
If you set this parameter to -1, the number is based on available CPU cores. | **Binary Format Parameters** You can modify these parameters to change the default behavior of the BINARY data format extraction. | **Binary Format Parameter Name** | **Default** | **Data Type** | **Required Value and Description** | | --------------------------------------------------------------- | ----------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sql.pipelineparameters.extract.binary.defaultEndianness` | `big` | string | The endianness of transforms when you do not specify it in another way.
Supported values are `big` or `little`. | | `sql.pipelineparameters.extract.binary.defaultCharset` | ibm1047 | string | Determines the default character set to use when you load data in the `BINARY` format. | | `sql.pipelineparameters.extract.binary.defaultAutoTrimPadding` | true | Boolean | Determines the default for whether a padding character should be trimmed when decoding data in the `BINARY` format to a string. | | `sql.pipelineparameters.extract.binary.defaultPaddingCharacter` | ' ' | `CHAR` | Determines the default padding character to trim from the end of a string when decoding data in the `BINARY` format.
This parameter is effective only if the `extract.binary.defaultAutoTrimPadding` parameter is set to `true`. | **Examples** Set the target execution time for the extractor task to 600 seconds. ```sql SQL theme={null} ALTER SYSTEM ALTER CONFIG SET 'sql.pipelineparameters.targetTimePerExtractorTask' 600; ``` Set the default endianness of transforms to little endian for the BINARY extract. ```sql SQL theme={null} ALTER SYSTEM ALTER CONFIG SET 'sql.pipelineparameters.extract.binary.defaultEndianness' 'little'; ``` The configuration values only go into effect after the `ALTER` SQL statement succeeds and you restart the affected nodes. ## Related Links [Load Data](/load-data) [Data Types for Data Pipelines](/data-types-for-data-pipelines) [Data Formats for Data Pipelines](/data-formats-for-data-pipelines) [Configuration Settings for Data Pipelines](/configuration-settings-for-data-pipelines) # Amazon Web Services Ocient Installation Source: https://docs.ocient.com/amazon-web-services-ocient-installation Install and deploy an Ocient System on Amazon Web Services (AWS), including EC2 instance selection, storage configuration, and networking for cloud workloads. This guide explains how to install an System in . For details about AWS concepts, see these pages: * [What is Amazon EC2?](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) * [What is IAM?](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) * [A Primer for Amazon Networking - VPC, Availability Zones & Subnets](https://aws.amazon.com/blogs/apn/amazon-vpc-for-on-premises-network-engineers-part-one/) Ocient supports deployment in AWS for pilot or testing purposes, but this setup does not guarantee data durability. Stopping instances can result in permanent data loss. The steps for deploying an Ocient System in AWS are: 1. Prepare AWS resources. 2. Set up an initial instance. 3. Create Amazon Machine Images (AMI) from the initial instance. 4. Launch other instances. 5. Follow the standard Ocient installation procedure. ## **Example Configuration** The table below shows the recommended instance types for each node type. | **Node Type** | **Instance Type** | | -------------------- | ----------------------- | | Foundation Nodes (3) | i3en.metal, i7ie.metal | | Loader Nodes (1) | i3en.metal, i7ie.metal | | SQL Nodes (1) | r5dn.metal, r6idn.metal | This diagram shows an example of an Ocient cluster in AWS. The EC2 nodes (SQL, Loader, and Foundation) are deployed within a single subnet of an Amazon . AWS assigns each type of node to a separate security group (`sg1`, `sg2`, `sg3`). Virtual private cloud for loading data from an S3 bucket using a JDBC client ## Prepare AWS Resources Create and configure these AWS resources: * The VPC and subnets for the Ocient System. * Security groups to access the endpoints for each node type. For details about the network security configuration, see the [Ocient Security Guide](/ocient-security-guide). * Identity and Access Management (IAM) roles. If you are loading data from S3, the loader nodes require IAM access to an S3 bucket. ## Node Setup (SQL Role) for Initial Instance Use this configuration for your AMI. Configuration steps differ depending on whether your setup uses a single-volume or multi-volume AMI. ### **Operating System (OS)** To set up the AMI, you can use any Ocient-supported OS (see [Ocient System Requirements](/ocient-system-requirements)). **Single-Volume AMI** If you use a single-volume AMI, specify this configuration: * Increase the root volume to 128GB or more. **Multi-Volume AMI** If you use a multi-volume AMI (e.g., CIS hardened (RHEL) 9), use this configuration: * Increase root volume to 30GB or more. * Increase Elastic Block Store (EBS) volume to 100GB or more. This EBS volume supports key system directories in the image (`/home`, `/var`, `/var/log`, `/var/log/audit`, `/var/tmp`). **Instance Type** Use r5dn.metal or a similar instance type. **Security Group** Use one or more security groups with these rules: * Allow SSH to the nodes. * Allow communication internally between nodes. * Allow access to SQL Node endpoints described in the [Ocient Security Guide](/ocient-security-guide). Connect to your instance using Secure Shell (SSH). For details, see [Connect to your Linux instance using an SSH client](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-linux-inst-ssh.html). This step applies only to multi-volume AMIs. If you are using a single-volume instance, skip this step. If you use a multi-volume manager, extend the `/home` and `/var` LVM volumes and their file systems to fill up the expanded EBS volumes. These actions expand the LVM volume and the contained file system to accommodate the Ocient package, logging, and metadata. These code examples show how to extend LVM volumes for a CIS RHEL 9 image. Other AMI types might require different sizing. Contact Ocient support for the best sizing for your system for multi-volume instances. **Example** Resize the physical volumes of two drives to use their full capacity after expanding them (see Step 1). ```shell Shell theme={null} sudo pvresize /dev/nvme0n1 sudo pvresize /dev/nvme1n1 ``` Extend local volumes: * Add 66 percent of the available free space in `vg-01` to the `var_vol` logical volume. * Add all (100 percent) of the remaining free space in `vg-01` to the `home_vol` logical volume. ```shell Shell theme={null} sudo lvextend -l +66%FREE /dev/vg-01/var_vol sudo lvextend -l +100%FREE /dev/vg-01/home_vol ``` Extend the file system to use all available space on its underlying logical volume. ```shell Shell theme={null} sudo xfs_growfs /home sudo xfs_growfs /var ``` Update all your software packages to their latest versions and then reboot your instance. For RHEL-compatible systems, use this command. ```shell Shell theme={null} sudo dnf update ``` For -compatible systems, use this command. ```shell Shell theme={null} sudo apt update ``` Reboot after the update. ```shell Shell theme={null} sudo reboot ``` Copy over the `ocient` RPM or DEB package and install it. For RHEL-compatible systems, use this command. ```shell Shell theme={null} sudo dnf install ./ocient-RELEASE-XX.X.X-xxxxxxxxxxxxxx-x86_64.rpm ``` For Debian-compatible systems, use this command. ```shell Shell theme={null} sudo apt install ./ocient-RELEASE-XX.X.X-xxxxxxxx.xxxxxx-xxxxxxxxxxxx-amd64.deb ``` Use the `ockernelparams` utility to set up kernel parameters automatically, including the huge pages configuration. Repeat this step on other nodes. ```shell Shell theme={null} sudo /opt/ocient/scripts/ockernelparams --node-role sql ``` Reboot the system for the parameters to take effect. ```shell Shell theme={null} sudo reboot ``` Check that the local storage drive is attached to the `uio` or `vfio` driver after reboot (the Ocient package installs a service that runs on startup to do this). ```shell Shell theme={null} sudo /opt/ocient/scripts/nvme-driver-util.sh ``` For examples of attaching drivers to the NVMe drives, see [NVMe Drive Firmware Upgrade Process](/nvme-drive-firmware-upgrade-process#page-title). If you are not using an OS-level firewall, skip this step. If your base AMI includes a system firewall, you must configure rules that explicitly allow required network communication for your Ocient deployment. For details, see [Ocient Security Guide](/ocient-security-guide). Required OS firewall rules: * Allow all necessary ports and protocols between Ocient nodes by: * Opening all TCP/UDP ports within the private network range (e.g., `10.0.0.0/16`). * Or, allowing known Ocient ports. * Allow external access where needed by: * Enabling SSH access (port 22) from your administrator IP range. * Allowing client access to SQL endpoints (for example, port 13101 or as specified in your setup). * Opening any additional ports required for monitoring or management tools. For a list of required ports, see [Network Exposure and Firewall](/ocient-security-guide#network-exposure-and-firewall). ## Create AMI After your initial node is fully configured, you must replicate the setup process for the remaining nodes in your cluster. This action ensures consistency and allows for proper internal communication between nodes. For details about creating an AMI, see [Creating an AMI from an Amazon EC2 Instance](https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html). ## Set Up Remaining Nodes to Launch the Remaining Instances Go through this process for each of your remaining nodes. Launch the remaining instances with these parameters: * AMI — Use the AMI created in the [Node Setup (SQL Role) for Initial Instance](#node-setup-sql-role-for-initial-instance) step. * Instance Type — Use `i3en.metal` or an equivalent instance that: * Offers local NVMe SSDs for high-performance local storage. * Has high throughput and network bandwidth for internal cluster communication. * Security groups — Ensure these security rules are in place in the AWS security groups associated with the nodes: * Allow SSH to the nodes. * Allow internal communication between all Ocient nodes. * Allow access to endpoints described in the [Ocient Security Guide](/ocient-security-guide). Connect to your instance using Secure Shell (SSH). For details, see [Connect to your Linux instance using an SSH client](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-linux-inst-ssh.html). Use the `ockernelparams` utility to set up kernel parameters automatically, including the `hugepages` parameters. This example specifies a Foundation Node `foundation`. Use a different node type as necessary. ```shell Shell theme={null} sudo /opt/ocient/scripts/ockernelparams --node-role foundation ``` ## Bootstrap the Ocient System Complete the bootstrapping process for your Ocient System. For details, see [Node Bootstrapping Reference](/node-bootstrapping-reference). ## Related Links [Load Data](/load-data) [Query Ocient](/query-ocient) # callback Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/get-callback openapi/generated-api-methods.json GET /v1/callback Provides the authentication token for the OpenID Connect authentication process. The authorization server redirects to callback path after successful authentication. This endpoint receives the authorization code from the OpenID provider and exchanges it for a token. The provider redirects the user to the application callback URL specified in the initial authentication request. The client application should not call this endpoint directly. The OpenID provider automatically calls this endpoint in the authentication process. # execute Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/get-execute openapi/generated-api-methods.json GET /v1/execute/{database} Alternative GET method for executing SQL statements. This method passes parameters as URL query parameters. This method does not support the params body parameter. Specify which database to access in the query parameters. If you do not specify a database, the connection defaults to the database from the authentication token or system settings. This method is most suitable for simple, read-only queries where the statement can be safely included in a URL. For complex queries or those with parameters, use the POST method instead. # info Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/get-info openapi/generated-api-methods.json GET /v1/info Returns basic system version information about the Ocient System and the HTTP Query API server. You can use this endpoint to verify connectivity and to check compatible versions. # execute Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/post-execute openapi/generated-api-methods.json POST /v1/execute/{database} Executes a SQL statement and returns the results. Supports requests for both regular and streaming responses. With streaming, results return as they become available using the HTTP chunked transfer encoding. Each chunk contains a valid JSON object that you can parse independently. For details on streaming, including large-response configuration, see the header parameters. You can specify the database in the request body. If you do not specify the database, the Ocient System defaults to the database specified in the authentication token or system settings. If you specify a database in the request, this value overrides any alternate database value specified as a body parameter. # login Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/post-login openapi/generated-api-methods.json POST /v1/login Authenticates a user with a username and password and then returns a token for use in subsequent API calls. This request also sets a session cookie. Include the returned access token in the authorization header for subsequent requests in the format: Authorization: Bearer {token}. # logout Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/post-logout openapi/generated-api-methods.json POST /v1/logout Log out from a SQL session. This clears any associated cookies, but does not invalidate any access tokens. This endpoint terminates only the cookie-based session. Any bearer tokens that were previously issued continue to work until they expire. You must discard any stored bearer tokens to complete the log out in a client application. # sso_authentication Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/post-sso-authentication openapi/generated-api-methods.json POST /v1/sso_authentication Initiates the OpenID Connect authentication process by redirecting to the authorization server. This endpoint begins the standard OpenID Connect authentication process: 1. The client calls this endpoint with a callback path. 2. The server responds with a redirect to the identity provider. 3. The user authenticates with the identity provider. 4. The identity provider redirects back to the callback endpoint. 5. The client can exchange the authorization code for an access token. # sso_device_grant Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/post-sso-device-grant openapi/generated-api-methods.json POST /v1/sso_device_grant Retrieve an OpenID device grant code that the Ocient System can verify and use with the sso_device_grant_verify endpoint. The device grant process is intended for devices with limited input capabilities or no web browser: 1. Call this endpoint to retrieve a user code and verification URI. 2. Display the user code and verification URI to the user. 3. The user visits the verification URI on another device and enters the code. 4. Call the sso_device_grant_verify endpoint to check if the user has completed verification. # sso_device_grant_verify Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/post-sso-device-grant-verify openapi/generated-api-methods.json POST /v1/sso_device_grant_verify Verify a previous device grant request and return an authorization token. After initiating a device grant process with the sso_device_grant endpoint, use this endpoint to check if the user has completed the verification process. If the verification is successful, the endpoint returns an authorization token that you can use for subsequent API calls. You can call this endpoint multiple times until one of these outcomes: The user completes the verification (returns 200 OK with a token). The timeout is reached (returns error). The verification is canceled (returns error). # sso_token Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/post-sso-token openapi/generated-api-methods.json POST /v1/sso_token Exchange an OpenID Connect identifier token or access token for an Ocient access token. This endpoint allows clients to directly exchange tokens without following the full browser-based authentication process. This exchange is useful for server-to-server scenarios or when the client already has a valid OpenID token from another process. # token_refresh Source: https://docs.ocient.com/api-playgrounds/ocient-http-query-api/post-token-refresh openapi/generated-api-methods.json POST /v1/token_refresh Refreshes an existing access token, extending its validity period. Call this endpoint before the current token expires to maintain uninterrupted access. Use the expires_in value from the login or previous refresh response to determine when to refresh the token. A common practice is to refresh when the token has half of its time remaining. On success, the endpoint returns the 200 response and a JSON object containing a new access token and metadata about the refreshed session. Replace the access token in your client with the new token returned by the endpoint, and use the expires_in value to decide when to refresh again. # Get Configuration Source: https://docs.ocient.com/api-playgrounds/system-information-rest-endpoints/get-configuration openapi/generated-api-methods.json GET /v1/sysconfig Retrieve the configuration about this node in JSON format. # Get Configuration of Whole System Source: https://docs.ocient.com/api-playgrounds/system-information-rest-endpoints/get-configuration-of-whole-system openapi/generated-api-methods.json GET /v1/dbconfig Retrieve the configuration of the system as a whole in JSON format. # Get Configuration Parameters Source: https://docs.ocient.com/api-playgrounds/system-information-rest-endpoints/get-configuration-parameters openapi/generated-api-methods.json GET /v1/config Retrieve values of configuration parameters. # Get Statistics Source: https://docs.ocient.com/api-playgrounds/system-information-rest-endpoints/get-statistics openapi/generated-api-methods.json GET /v1/stats Retrieve the statistics on each node in the database. These fields appear in every entry returned in the response from the `http://oc1-lts0:9090/v1/stats` endpoint: * `name` * `time` * `timestamp` * `node` * `value` Additional fields can vary. # Get Status Source: https://docs.ocient.com/api-playgrounds/system-information-rest-endpoints/get-status openapi/generated-api-methods.json GET /v1/status Retrieve the status of the software running. # Get Version Source: https://docs.ocient.com/api-playgrounds/system-information-rest-endpoints/get-version openapi/generated-api-methods.json GET /v1/version Retrieve the version of the software running. # Array Functions and Operators Source: https://docs.ocient.com/array-functions-and-operators Reference for Ocient SQL array functions and operators, with examples for constructing, accessing, slicing, concatenating, and aggregating array values. The System enables you to work with data as arrays. The system contains functions that work with arrays and operators that enable you to parse data within arrays. ## ARRAY\_CAT\_DISTINCT Concatenates two or more arrays in the order of the input arguments. After concatenation, the function removes duplicates from the result array while preserving the first occurrence order. The function skips any NULL input arguments. If all input arrays are NULL, the function returns a NULL. If at least one NULL element is present in the array, the function returns a NULL in the result array with the position of the first NULL the function finds in the order of the input arguments. **Syntax** ```sql SQL theme={null} ARRAY_CAT_DISTINCT(input_array1, input_array2 [, ...]) ``` | **Argument** | **Data** **Type** | **Description** | | ------------------------------------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------- | | `input_array1, input_array2 [, ...]` | ARRAY | Two or more arrays for concatenation and deduplication. The elements within all input arrays must have the same data type. | **Examples** **Concatenate Two Arrays and Remove Duplicates** Concatenate two arrays and remove duplicate elements within the arrays. ```sql SQL theme={null} SELECT ARRAY_CAT_DISTINCT(ARRAY[1,2,2,3,1], ARRAY[2,3,4,4,5]); ``` Output: `[1,2,3,4,5]` **Concatenate Two Arrays with NULLs and Remove Duplicates** Deduplicate the contents of the array. In this case, the array contains NULLs. ```sql SQL theme={null} SELECT ARRAY_CAT_DISTINCT(ARRAY[1,2,2,NULL,1], ARRAY[2,4,4,NULL]); ``` Output: `[1,2,NULL,4]` ## ARRAY\_DISTINCT Removes duplicates from an array while preserving the first occurrence order. If at least one NULL is present in the array, the function returns a NULL with the position of the first NULL of the input array in the output array. **Syntax** ```sql SQL theme={null} ARRAY_DISTINCT(input_array) ``` | **Argument** | **Data** **Type** | **Description** | | ------------- | ----------------- | ---------------------------- | | `input_array` | ARRAY | The array for deduplication. | **Examples** **Remove Duplicates from an Array** Deduplicate the contents of the array. ```sql SQL theme={null} SELECT ARRAY_DISTINCT(ARRAY[1,2,2,3,1]); ``` Output: `[1,2,3]` **Remove Duplicates from an Array with NULLs** Deduplicate the contents of the array. In this case, the array contains NULLs. ```sql SQL theme={null} SELECT ARRAY_DISTINCT(ARRAY[1,2,NULL,2,3,NULL,1]); ``` Output: `[1,2,NULL,3]` ## ARRAY\_LENGTH Returns the length of the array for the specified dimension. If the array is a nested array, the function returns the length of the outermost array. **Syntax** ```sql SQL theme={null} ARRAY_LENGTH(input_array) ``` | **Argument** | **Data** **Type** | **Description** | | ------------- | ----------------- | ------------------------------------------------------------------------------------------- | | `input_array` | ARRAY | The array that contains any type of elements. The array can be single or multi-dimensional. | **Example** Return the length of the array. ```sql SQL theme={null} SELECT ARRAY_LENGTH(ARRAY[1,2]); ``` Output: `2` ## ARRAY\_SORT Sorts and returns the input array based on the natural ordering of its elements. The behavior of this function varies depending on the syntax you use. **Syntaxes** **Basic Array Sort** Sorts and returns the input array based on the natural ordering of its elements. If one of the array elements is NULL, then this function sorts the NULL values to the end of the array. ```sql SQL theme={null} ARRAY_SORT(input_array) ``` | **Argument** | **Data** **Type** | **Description** | | ------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `input_array` | ARRAY | An array of elements with these types: `BIGINT`, `BOOLEAN`, `BYTE`, `DATE`, `DOUBLE`, `FLOAT`, `INT`, `SMALLINT`, `UUID`, or `VARCHAR`. | **Sort an Array With a Lambda Function** Sorts and returns the input array based on the results of the specified Lambda function. The function should have two arguments representing two elements of the array. This function should return a negative integer, 0, or a positive integer if the first element is less than, equal to, or greater than the second element, respectively. ```sql SQL theme={null} ARRAY_SORT(input_array,function) ``` | **Argument** | **Data** **Type** | **Description** | | ------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------- | | `input_array` | ARRAY | An array that contains any type of elements. The array can be single or multi-dimensional. | | `function` | FUNCTION | This is an optional argument. A lambda function with the format `(x T, y T) -> INT` or another SQL reference function. | The Ocient System supports this syntax with the data pipeline functionality only. For details, see [Transform Data in Data Pipelines](/transform-data-in-data-pipelines). **Sort an Array in the Specified Order** Determines how the array handles the order of elements in the array. ```sql SQL theme={null} ARRAY_SORT(input_array,sort_order) ``` | **Argument** | **Data** **Type** | **Description** | | ------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `input_array` | ARRAY | An array of elements with these types: `BIGINT`, `BOOLEAN`, `BYTE`, `DATE`, `DOUBLE`, `FLOAT`, `INT`, `SMALLINT`, `UUID`, or `VARCHAR`. | | `sort_order` | BOOLEAN | This is an optional argument.

Determines the sort order of the array. The value `true` means ascending order. The value `false` means descending order. The default value is `true`. | **Sort an Array in the Specified Order with NULL Elements** Determines how the array handles the order of elements in the array and where NULL elements appear. ```sql SQL theme={null} ARRAY_SORT(input_array,sort_order,nulls_first) ``` | **Argument** | **Data** **Type** | **Description** | | ------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `input_array` | ARRAY | An array of elements with these types: `BIGINT`, `BOOLEAN`, `BYTE`, `DATE`, `DOUBLE`, `FLOAT`, `INT`, `SMALLINT`, `UUID`, or `VARCHAR`. | | `sort_order` | BOOLEAN | This is an optional argument.

Determines the sort order of the array. The value `true` means ascending order. The value `false` means descending order. The default value is `true`. | | `nulls_first` | BOOLEAN | This is an optional argument.

Determines whether NULL elements appear first or last in the array. The `true` value means that NULLs appear first in the array. The `false` value means that they appear last. If the array is in ascending order, the default value is `false`. Otherwise, the default value is `true`. | **Examples** **Sort an Array** Sort an array of three elements `[2,3,1]`. ```sql SQL theme={null} SELECT ARRAY_SORT(ARRAY[2,3,1]); ``` Output: `[1,2,3]` **Sort an Array by Specifying the Sort Order** Sort an array of four elements `[2,3,1,NULL]` and specify the order of the elements in descending order. ```sql SQL theme={null} SELECT ARRAY_SORT(ARRAY[2,3,1,NULL],false); ``` Output: `[NULL,3,2,1]` **Sort an Array by Specifying the Sort Order and NULL Placement** Sort an array of four elements `[2,NULL,3,1]` and specify the order of the elements in ascending order, and have the NULL element appear last. ```sql SQL theme={null} SELECT ARRAY_SORT(ARRAY[2,NULL,3,1],true,false); ``` Output: `[1,2,3,NULL]` ## UNNEST Expands each element in an input array into an individual row. For example, the `UNNEST` function on an array column of type `ARRAY(INT)` with values `[2, 6]` yields two result rows with integers `2` and `6`. The values of the other columns in each input row are unchanged in each corresponding output row. You can specify multiple array columns to unnest the specified arrays from each row in parallel. **Syntax** ```sql SQL theme={null} UNNEST( array_column [, ... ]​ ) [ WITH ] ::= { ORDINALITY [ ord_identifier ] | VALUE [ val_identifier ] | NULL_INPUT } ``` | **Argument** | **Data** **Type** | **Description** | | ---------------- | ----------------- | ----------------------------------------------------------------------------------------------------- | | `array_column` | `ARRAY` | The array column to expand. | | `ord_identifier` | `CHAR` | When you use this argument in an `ORDINALITY` clause, the argument is the name for the output column. | | `val_identifier` | `CHAR` | When you use this argument in a `VALUE` clause, the argument is the name for the unnested column. | The `UNNEST` function supports these options. When you use only one of these options, enclosing the option in parentheses is optional. However, if you use more than one of these options, you must enclose them in parentheses. | **Options** | **Description** | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ORDINALITY [ord_identifier]` | Optional.
Generates a new integer column that corresponds to the index position of each output element in the original array. The `ord_identifier` value specifies the name of the output ordinality column.
If you do not specify this clause, the column name defaults to the original column name with the suffix `_ord`. | | `VALUE [val_identifier]` | Optional.
Rename the unnested column to the new name specified in the `val_identifier` argument.
If you do not specify this clause, the column name defaults to the original column name with the suffix `_val`. | | `NULL_INPUT` | Optional.
When the input column array is empty or NULL, this clause specifies to replace each output row with an array with one NULL element. This replacement preserves the contents of other input rows. By default, this option is not enabled. | These examples demonstrate how the `UNNEST` function operates on arrays and other objects. Most of the examples use the data rows from this table. ```Text Text theme={null} a b c d e f -------------------------------------------- 1 [1, 2] [6, 9] [] NULL [[1, 2], [5], []] 2 [7, 5] [4] [3] [8] [[6], [9, 4, 7], [3, 8]] ``` **Examples** **Expand a Column** This example performs a simple `UNNEST` on column `b`. ```sql SQL theme={null} SELECT UNNEST(b); ``` *Output* ```Text Text theme={null} b_val ----------- 1 2 7 5 ``` **Expand a Column Using the** `ORDINALITY` **Option** In this example, the `ORDINALITY` option adds a second return column that represents the index position of the value in the input array. ```sql SQL theme={null} SELECT UNNEST(b) WITH ORDINALITY; ``` *Output* ```Text Text theme={null} b_val b_ord -------------------- 1 1 2 2 7 1 5 2 ``` **Expand a Column with Empty Data** In this example, the `UNNEST` query returns only one row because one of the two rows in column `d` is an empty array. If you use the `NULL_INPUT` option, the query returns a second row with the `NULL` value. ```sql SQL theme={null} SELECT UNNEST(d); ``` *Output* ```Text Text theme={null} d_val ------------ 3 ``` **Expand a Column with NULL Data** `UNNEST` also does not return array values that are `NULL` unless you specify the `NULL_INPUT` option. ```sql SQL theme={null} SELECT UNNEST(e); ``` *Output* ```Text Text theme={null} e_val ------------ 8 ``` **Expand a Column with Multiple Array Layers** The `UNNEST` function expands only one array layer. The empty array remains in the returned values. ```sql SQL theme={null} SELECT UNNEST(f); ``` *Output* ```Text Text theme={null} f_val ------------ [1,2] [5] [] [6] [9,4,7] [3,8] ``` **Expand Columns Using Multiple** `UNNEST` **Statements** Multiple `UNNEST` statements in a query produce a per-row Cartesian product of all the unnested values. ```sql SQL theme={null} SELECT UNNEST(b), UNNEST(c); ``` *Output* ```Text Text theme={null} b_val c_val ------------ 1 6 1 9 2 6 2 9 7 4 5 4 ``` **Expand Columns Using Multiple** `UNNEST` **Statements in Reverse Order** This example uses `UNNEST` on the same columns but in reverse order. ```sql SQL theme={null} SELECT UNNEST(c), UNNEST(b); ``` *Output* ```Text Text theme={null} c_val b_val ------------ 6 1 6 2 9 1 9 2 4 7 4 5 ``` **Expand Multiple Columns** In this example, the query unnests two array columns in parallel. Even though it does not include the `NULL_INPUT` option, the query still returns NULL for column `c` to correspond with column `b`. ```sql SQL theme={null} SELECT UNNEST(b, c); ``` *Output* ```Text Text theme={null} b_val c_val ------------ 1 6 2 9 7 4 5 NULL ``` **Expand Multiple Array Columns** This query unnests two array columns. ```sql SQL theme={null} SELECT UNNEST(b, d); ``` *Output* ```Text Text theme={null} b_val d_val ------------ 1 NULL 2 NULL 7 3 5 NULL ``` **Expand Values of Arrays Within Arrays** This example has two layers of `UNNEST` to capture values of arrays within arrays. ```sql SQL theme={null} SELECT UNNEST(f_val) FROM (UNNEST(f)); ``` *Output* ```Text Text theme={null} f_val ------------ 1 2 5 6 9 4 7 3 8 ``` **Expand an Empty Array** This example unnests an empty integer array. ```sql SQL theme={null} SELECT UNNEST(INT[]()); ``` *Output* ```Text Text theme={null} col_val ------------ ``` The query returns a column of type `INT` but no rows. **Expand an Empty Array Column Without the** `NULL_INPUT` **Option** This example selects the non-array column `a` and an empty array column. The query returns no rows because empty or `NULL` array values are not returned unless you specify the `NULL_INPUT` option. ```sql SQL theme={null} SELECT a, UNNEST(INT[]()); ``` *Output* ```Text Text theme={null} a col_val ------------ ``` **Expand a Column Without the** `NULL_INPUT` **Option** This query does not include the `NULL_INPUT` option, which causes the result to omit the NULL array values in column `e`. ```sql SQL theme={null} SELECT a, UNNEST(e); ``` *Output* ```Text Text theme={null} a e_val ------------ 1 8 2 8 ``` **Expand a Column** **with the** `NULL_INPUT` **Option** This query includes the `NULL_INPUT` option, which means that the query returns the NULL values in column `e`. ```sql SQL theme={null} SELECT a, UNNEST(e) WITH NULL_INPUT; ``` *Output* ```Text Text theme={null} a e_val ------------ 1 8 1 NULL 2 8 2 NULL ``` **Expand Arrays with NULL Rows** This example attempts to unnest two arrays, one of which is cast as NULL, while the other is empty. In both cases, the query returns a column of type `INTEGER` with no rows because there are no values to unnest. ```sql SQL theme={null} SELECT UNNEST(CAST(NULL AS INT[]); SELECT UNNEST(INT[]()); ``` *Output* ```Text Text theme={null} col_val ------------ ``` ## Other Array Functions | **Function** | **Syntax** | **Purpose** | | ------------------------ | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Constructor | TYPE\[]\(e1, e2, …, en) | Construct an array of SQL type TYPE giving the elements. NULL is also supported as an element. | | Constructor (2) | array\[e1, e2, …, en] | -compliant constructor. The type of the array is deduced from the elements. | | array concatenation | array\_cat(array, array) | Concatenate 2 arrays into a new one. | | array softmax | softmax(array) | Returns the softmax of the array. The array must be one-dimensional and contain float or double values. The system transfers NULL-valued indices as 0.0. | | array cross entropy loss | cross\_entropy\_loss(array, array) | Returns the cross entropy loss of two arrays. Both arrays must be one-dimensional and contain float or double values. NULL values do not contribute to the sum. | | array log loss | log\_loss(array, array) | Returns the log loss of two arrays. Both arrays must be one-dimensional and contain float or double values. NULL values do not contribute to the sum. | | array logits loss | logits\_loss(array, array) | Returns the logits loss of two arrays. Both arrays must be one-dimensional and contain float or double values. NULL values do not contribute to the sum. | | array hinge loss | hinge\_loss(array, array) | Returns the hinge loss of two arrays. Both arrays must be one-dimensional and contain float or double values. NULL values do not contribute to the sum. | | array sum | array\_sum(array) | Returns the sum of the array. The array must be one-dimensional and contain numeric values. NULL values do not contribute to the sum. | | array min/max | array\_min(array), array\_max(array) | Returns the corresponding minimum or maximum of the array. This function only considers the first dimension of the array. If the array contains only NULL values, then the function returns NULL. | | array argmax | array\_argmax(array) | Returns the 1-based position (index) of the maximum element in an array. The first element has index 1.

This function only considers the first dimension of the array. If the array contains only NULL values, then the function returns NULL. | | array argmin | array\_argmin(array) | Returns the 1-based position (index) of the minimum element in an array. The first element has index 1.

This function only considers the first dimension of the array. If the array contains only NULL values, then the function returns NULL. | | array casting | cast\_to\_array(array, format) | Casts the elements of the array to another type, as specified by the format string. The format is: `'ARRAY(INT)'` for an `int` array, `'ARRAY(CHAR)'` for a `char` array, and so on. Currently, only numeric casts are supported. | | scalar position | array\_position(array, scalar, pos = 1 ) | Returns the position of the first matching scalar in the array. pos indicates the position to start searching, if not specified it defaults to 1. | | array elements positions | array\_positions(array, array ) | Same as array\_position(). But this looks for all elements stored in the right array and returns their respective positions in the left array. | | array append | array\_append(array, value) | Add value to the back of an array | | array prepend | array\_prepend(value, array) | Add value to the front of an array | | array remove | array\_remove(array, value) | remove value from the array | | array replace | array\_replace(array, old\_value, new\_value) | replace value by another in an array | | convert array to string | char(array) | Converts array to its string representation. | | convert array to string | array\_to\_string(array, delimiter \[, nullstr]) | Converts array to a list of elements separated by 'delimiter'. Nested arrays are flattened. NULL values are either ignored or replaced by *nullstr* if provided. | | convert string to array | string\_to\_array(str, format) | Converts the string representation of an array (e.g., `'int[1,2,NULL]'`) into an array. The format string is the same as in cast\_to\_array. | ### Function Examples | **Function** | **SQL statement** | **Result** | | ------------------------ | ------------------------------------------------ | ----------------------------------- | | type constructor | int\[]\(1,2) | int array with values 1 and 2 | | constructor with no type | array\[1,2] | bigint array with values 1 and 2 | | array\_cat | array\_cat(array\[2, 3], array\[4, 5]) | array\[2, 3, 4, 5] | | array\_length | array\_length(array\[2, 3]) | 2 | | cast\_to\_array | cast\_to\_array(array\[2.5, 3.4], 'ARRAY(INT)') | int\[]\(2,3) | | array\_position | array\_position(array\[1, 2, 1], 1) | 1 | | | array\_position(array\[1, 2, 3, 4, 1], 1, 3) | 5 | | array\_positions | array\_positions(array\[1, 2, 3, 4, 1], 1) | int\[]\(1,5) | | array\_append | array\_append(array\[1,2], 3) | array\[1,2,3] | | array\_prepend | array\_prepend(0, array\[1,2]) | array\[0,1,2] | | array\_remove | array\_remove(array\[1,2], 1) | array\[2] | | array\_replace | array\_replace(array\[1,2,4],4,3) | array\[1,2,3] | | softmax | softmax(float\[]\(2.0, 2.0, 2.0, NULL, 2.0)) | float\[0.25, 0.25, 0.25, 0.0, 0.25] | | array\_sum | array\_sum(int\[]\(1, 2, 3)) | 6 | | array\_min | array\_min(int\[]\(1, -2, 3)) | -2 | | array\_argmax | array\_argmax(int\[]\(1, 3, 2)) | 2 | | array\_argmin | array\_argmin(int\[]\(1, 3, 2)) | 1 | | char(\) | char(array\[1,2]) | 'bigint\[1,2]' | | array\_to\_string | array\_to\_string(int\[1,2,3,NULL,5], ',', '\*') | '1,2,3,\*,5' | | string\_to\_array | string\_to\_array('int\[1,2]', 'ARRAY(INT)') | int\[]\(1,2) | ## Array Operators Ocient array operators allow you to concatenate and check for containment or overlap of data. Also, you can retrieve specific elements or slices within arrays. Do not use array operators, including `@>`, `<@`, and `&&`, to evaluate `NULL` values in arrays. For information on how to check for `NULL` values in arrays, see [Array NULL Handling](#array-null-handling). ### Contains Operator (`@>`) The `@>` operator determines whether a left-side array contains a scalar value or array elements on the right side. `@>` **Syntax** ```sql SQL theme={null} array @> scalar_or_array ``` | **Argument** | **Data** **Type** | **Description** | | ----------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `array` | `ARRAY` | An array column or expression. | | `scalar_or_array` | `ARRAY` or any numeric, character, time, or geospatial data type. | A value or an array of values to check whether they are contained in the array. For data types, see [Understanding Data Types](/understanding-data-types). | **Examples** These examples use the `@>` operator to test whether the left-side array contains all the elements from the right-side array. **Array Containment (True Case)** This example returns `true` because all right-side elements are in the left-side array. ```sql SQL theme={null} SELECT ARRAY[1, 4, 3] @> ARRAY[3, 1]; ``` *Output:* `true` **Array Containment (False Case)** If the right-side array contains at least one value not present in the left-side array, the query returns `false`. In this example, the right-side array has one value, `5`, not present on the left side. ```sql SQL theme={null} SELECT ARRAY[1, 4, 3] @> ARRAY[3, 1, 5]; ``` *Output:* `false` **Scalar Containment in an Array** This example checks whether a single scalar value is in the left-side array. ```sql SQL theme={null} SELECT ARRAY[3, 1, 3] @> 1; ``` *Output:* `true` ### Contained In Operator (`<@`) The `<@` operator checks whether a right-side array contains all the elements on the left side. **Syntax** ```sql SQL theme={null} scalar_or_array <@ array ``` | **Argument** | **Data** **Type** | **Description** | | ----------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `array` | `ARRAY` | An array column or expression. | | `scalar_or_array` | `ARRAY` or any numeric, character, time, or geospatial data type. | A value or an array of values to check whether they are contained in the array. For data types, see [Understanding Data Types](/understanding-data-types). | **Examples** **Array Contained Within Another Array (True Case)** This example returns `true` because all left-side elements are in the right-side array. ```sql SQL theme={null} SELECT VARCHAR[]('apples', 'oranges') <@ VARCHAR[]('apples', 'oranges', 'bananas') ``` *Output:* `true` **String Membership Check** `<@` and other array operators can check whether individual strings are present in an array. The `'oranges'` string is present in the right-side array. ```sql SQL theme={null} SELECT 'oranges' <@ VARCHAR[]('apples', 'oranges', 'bananas') ``` *Output:* `true` ### Overlap Operator (`&&`) The overlap operator `&&` determines whether any elements between two arrays are common. **Syntax** ```sql SQL theme={null} array1 && array2 ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------ | | `array1` | `ARRAY` | An array column or expression to check whether any of its values are shared with `array2`. | | `array2` | `ARRAY` | An array column or expression to check whether any of its values are shared with `array1`. | **Examples** **Check Overlap Between Arrays (True Case)** As long as at least one value is present in both arrays, the `&&` operator returns `true`. Both of these arrays contain the value `3`. ```sql SQL theme={null} SELECT ARRAY[1 ,2, 3] && ARRAY[3 ,4, 5]; ``` *Output:* `true` **Check Overlap Between Arrays (False Case)** Both of these arrays have no values in common, so the `&&` operator returns `false`. ```sql SQL theme={null} SELECT ARRAY[1 ,2, 3] && ARRAY[4 ,5, 6]; ``` *Output:* `false` ### Slice Operator (`:`) The slice operator `:` returns a subarray ranging from a left index to a right index, both of which are optional to specify. If you exclude both indexes, the slice operator returns the full array. **Syntax** ```sql SQL theme={null} array[left_index:right_index] ``` | **Argument** | **Data** **Type** | **Description** | | ------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `array` | `ARRAY` | An array column or expression. | | `left_index` | `INT` | Optional.

A starting index for the slice operator to begin the subarray. Index values start at `1`.

If you do not specify a left index, or if this value is less than zero, then the slice starts at index `1`. | | `right_index` | `INT` | Optional.

An ending index for the slice operator to end the subarray. Index values start at `1`.

If you do not specify a right index or if the value exceeds the array length, the right index defaults to the array length. | The slice operator also follows these rules: * Each index starts at `1`. * If the array or either index value is NULL, the result of slicing is NULL. * Ranges completely out of array bounds return an empty array. For example, `array[1, 2][4:5] = []`. * Sequential slices slice each dimension of multidimensional arrays. For example, `ARRAY[ARRAY[1, 2, 3], ARRAY[4, 5, 6]][1:1][2:3] = ARRAY[ARRAY[2,3]]`. * More than N sequential slices of an N-dimensional array, for example, three sequential slices on a two-dimensional array return an empty array. For example, `ARRAY[1, 2, 3][:][:] = []`. * You cannot combine slicing with the access operator when slicing multidimensional arrays. Any access operator `[n]` converts to `[:n]`. For example, `Array_Val[4][1:6]` would be equivalent to `Array_Val[:4][1:6]`. * Slicing an array of tuples such as `ARRAY[TUPLE<>(1,2), TUPLE<>(3,4)][1:1][2:3]` slices `[1:1]` on the array and `[2:3]` on the tuple elements within the sliced array, and returns the value `TUPLE<>[TUPLE<>(2)]`. **Examples** **Basic Array Slicing** Slice an array with four numbers starting at index `2` and ending at index `3`. ```sql SQL theme={null} SELECT ARRAY[1 ,2, 3, 4][2:3]; ``` \*Output: \*`['2','3']` **Array Slicing With No Right Index** The query returns all values after the second value because it does not specify an ending index. ```sql SQL theme={null} SELECT ARRAY[1 ,2, 3, 4][2:]; ``` \*Output: \*`['2','3','4']` **Array Slicing With No Left Index** This query captures a subarray starting at the first index because it does not specify a starting value. ```sql SQL theme={null} SELECT ARRAY[1 ,2, 3, 4][:3]; ``` \*Output: \*`['1','2','3']` ## Array NULL Handling Filtering with array functions can have different outcomes if they operate on an array containing NULL values or a NULL value of array type. Array comparison operators, such as `@>`, `<@`, and `&&`, do not follow normal Boolean logic when evaluating NULL values. To evaluate arrays for NULL values, use the filter functions `FOR_ALL()` or `FOR_SOME()`. For details about these functions, see [Array Filters](/data-query-language-dql-statement-reference#array-filters). **Examples** **Evaluate a Single Array With No NULL Values** This example evaluates to `false` because none of the array values are NULL. ```sql SQL theme={null} SELECT FOR_SOME(ARRAY[1, 2, 3]) IS NULL; ``` \*Output: \*`false` **Evaluate a Single Array With NULL Values** This example evaluates to `true` because the array contains a NULL value. ```sql SQL theme={null} SELECT FOR_SOME(ARRAY[1, 2, NULL]) IS NULL; ``` \*Output: \*`true` **Select Only Arrays Containing NULL Values** To further demonstrate array NULL behavior, these examples use this table loaded with a few array values, some of which are NULL or contain NULL values. ```sql SQL theme={null} CREATE TABLE demo_array_table ( id INT, tags VARCHAR[] ); INSERT INTO demo_array_table (id, tags) VALUES (1, VARCHAR[]('alpha', 'beta', 'gamma')), -- Regular array (2, VARCHAR[]('delta', NULL, 'epsilon')), -- Array containing a NULL element (3, NULL), -- Entirely NULL array (4, VARCHAR[]()), -- Empty array (5, VARCHAR[](NULL, NULL)); -- Array containing all NULL elements ``` In this example, the query uses a `FOR_SOME` function to filter the rows to return only arrays with NULL values. The output does not include the third row of the table because the row itself is NULL, and is not an array containing NULL values. ```sql SQL theme={null} SELECT id, tags FROM demo_array_table WHERE FOR_SOME(tags) IS NULL; ``` *Output* ```sql SQL theme={null} | id | tags | |----|----------------------------------| | 2 | ['delta', NULL, 'epsilon'] | | 5 | [NULL, NULL] | ``` **Select NULL Values or Arrays Containing NULL Values** This example includes an additional filter to include any NULL rows and arrays with NULL values. The example uses the `demo_array_table` table, which has values that are NULL or contain NULL values. ```sql SQL theme={null} SELECT id, tags FROM demo_array_table WHERE FOR_SOME(tags) IS NULL OR tags IS NULL; ``` *Output* ```sql SQL theme={null} | id | tags | |----|------------------------------| | 3 | NULL | | 5 | [NULL, NULL] | | 2 | ['delta', NULL, 'epsilon'] | ``` **Select Arrays Without NULL Values** This example filters the rows using the `FOR_ALL` filter function to return only arrays with no NULL values. The output also contains rows with empty arrays. The example uses the `demo_array_table` table, which has values that are NULL or contain NULL values. ```sql SQL theme={null} SELECT id, tags FROM demo_array_table WHERE FOR_ALL(tags) IS NOT NULL; ``` *Output* ```sql SQL theme={null} | id | tags | |----|-------------------------------| | 4 | [] | | 1 | ['alpha', 'beta', 'gamma'] | ``` **Select Only Non-Empty Arrays Without NULL Values** To omit empty arrays, you can add a filter to check the array length. The `ARRAY_LENGTH` function removes any empty arrays. This example uses the `demo_array_table` table, which has values that are NULL or contain NULL values. ```sql SQL theme={null} SELECT id, tags FROM demo_array_table WHERE FOR_ALL(tags) IS NOT NULL AND ARRAY_LENGTH(tags) > 0; ``` *Output* ```sql SQL theme={null} | id | tags | |----|-------------------------------| | 1 | ['alpha', 'beta', 'gamma'] | ``` ## Multidimensional Arrays Multidimensional arrays are supported. As seen in these examples, each dimension, or length, at each level does not have to be the same. NULL values are allowed at each dimension of an array. **Multidimensional Examples** | **Function** | **SQL statement** | **Result** | | -------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------ | | type constructor two-dimensional | `int[][](int[](1, NULL), array[1])` | two-dimensional `INT` array with values `[1, NULL]` and `[1]` | | constructor with no type two-dimensional | `array[int[]\(1), array[int(2)]]` | two-dimensional `INT` array with values `[1]` and `[2]` | | two-dimensional array with NULL values | `array[array[1, NULL], NULL]` | two-dimensional `BIGINT` array with values `[1, NULL]` and `NULL` | | two-dimensional array with different lengths | `array[array[1], array[1,2], array[1,2,3]]` | two-dimensional `BIGINT` array with values `[1]`, `[1,2]`, and `[1,2,3]` | ## Related Links [Matrix Functions and Operators](/matrix-functions-and-operators) [Tuple Functions and Operators](/tuple-functions-and-operators) [Math Functions and Operators](/math-functions-and-operators) [Array Data Transformation Functions](/transform-data-in-data-pipelines#array-data-transformation-functions) [Data Types](/data-types) [Data Types for Data Pipelines](/data-types-for-data-pipelines) [Query Ocient](/query-ocient) # Array, Tuple, and Matrix Overview Source: https://docs.ocient.com/array-tuple-and-matrix-overview Overview of array, tuple, and matrix data structures in Ocient SQL, with guidance on constructing, indexing, and operating on nested or composite values. Arrays, tuples, and matrix data types are complex data types in the System. ## Arrays The array data type is a one-dimensional list of values with the same data type. This type enables the support of a one-to-many relationship without creating another table. You create secondary indexes on arrays, which index the elements of the array. For details about secondary indexes, see [Secondary Indexes](/secondary-indexes). Key features: * Variable size. * Can contain many elements. * Indexing into the array starts at `1`. * Invalid access of the array returns NULL. For semi-structured data in JSON format, you can express the JSON multi-value column in an array data type. Traditionally in the normalized relational data model, the database represents such data types as a separate table. Each row stores the value of the primary key of the record in the parent table. Therefore, the entity relationship between the tables is always one-to-many, which can result in billions of rows. The array data type eliminates the child tables, which helps reduce the storage footprint and avoids the additional JOIN operations. **Examples** Create a one-dimensional integer array `[1, 2, 3]` using the `sys.dummy` virtual table. ```sql SQL theme={null} SELECT INT[](1,2,3) FROM sys.dummy1; ``` *Output* ```none Text theme={null} array_int(int((1)), int((2)), int((3))) -------------------------------------------------------------------------------- [1,2,3] Fetched 1 row ``` Create a two-dimensional integer array `[[1, 2, 3], [4, 5, 6]]`. ```sql SQL theme={null} SELECT INT[][](INT[](1,2,3),INT[](4,5,6)) FROM sys.dummy1; ``` *Output* ```none Text theme={null} array_array(int)(array_int(int((1)), int((2)), int((3))), array_int(int((4)), int((5)), int((6)))) --------------------------------------------------------------------------------------------------- [[1,2,3],[4,5,6]] Fetched 1 row ``` You can get the same result by casting the array and integers explicitly. ```sql SQL theme={null} SELECT INT[][](ARRAY[INT(1),INT(2),INT(3)],ARRAY[INT(4),INT(5),INT(6)]) FROM sys.dummy1; ``` For more examples, see [Array Functions and Operators](/array-functions-and-operators). ## Tuples The tuple data type is a row. This type is a fixed-sized collection of heterogeneous values. Tuples support more complex recursive computational functions. The memory structure of the tuple is identical to the memory structure of the array type. The only difference is that each entry in a tuple can be any type, so the Ocient System can simultaneously store both fixed and variable-sized values in a tuple. **Examples** Create a tuple with fixed-size types. In this case, use three integers. ```sql SQL theme={null} SELECT TUPLE(1,2,3) FROM sys.dummy1; ``` *Output* ```none Text theme={null} tuple((1), (2), (3)) -------------------------------------------------------------------------------- <<1, 2, 3>> Fetched 1 row ``` You can get the same result by using the tuple type constructor. ```sql SQL theme={null} SELECT TUPLE<>(1,2,3) FROM sys.dummy1; ``` Create a tuple with variable-size types. In this case, use a one-dimensional and two-dimensional array. ```sql SQL theme={null} SELECT TUPLE(ARRAY[INT(1),INT(2),INT(3)], INT[][](ARRAY[INT(1),INT(2),INT(3)],ARRAY[INT(4),INT(5),INT(6)])) FROM sys.dummy1; ``` *Output* ```none Text theme={null} tuple(array_int(int((1)), int((2)), int((3))), array_array(int)(array_int(int((1)), int((2)), int((3))), array_int(int((4)), int((5)), int((6))))) --------------------------------------------------------------------------------------------------------------------------------------------------- <<[1,2,3], [[1,2,3],[4,5,6]]>> Fetched 1 row ``` Create a tuple with different types. In this case, an integer, string, and array of integers. ```sql SQL theme={null} SELECT TUPLE<>(1, 'test_string', INT[](2,4,6)) FROM sys.dummy1; ``` *Output* ```none Text theme={null} tuple(int((1)), ('test_string'), array_int(int((2)), int((4)), int((6)))) -------------------------------------------------------------------------------- <<1, test_string, [2,4,6]>> Fetched 1 row ``` For more examples, see [Tuple Functions and Operators](/tuple-functions-and-operators). ## Matrices A matrix is a fixed-size two-dimensional array of `DOUBLE` values. A matrix can be a row vector (1xN matrix) or column vector (Nx1 matrix). matrices support machine learning model calculations. For details, see [Machine Learning in Ocient](/machine-learning-in-ocient). **Examples** Create a 1x4 matrix with numbers 1 through 4 using the `make_matrix` function that expects values in row-major order. The first two arguments indicate the dimensions of the matrix, and the remaining arguments specify the values for the matrix. ```sql SQL theme={null} SELECT make_matrix_1x4(1,4,1,2,3,4) FROM sys.dummy1; ``` *Output* ```none Text theme={null} make_matrix_1x4((1), (4), (1), (2), (3), (4)) -------------------------------------------------------------------------------- [[1.0,2.0,3.0,4.0]] Fetched 1 row ``` Create a 2x3 matrix with numbers 1 through 6. ```sql SQL theme={null} SELECT make_matrix_2x3(2,3,1,2,3,4,5,6) FROM sys.dummy1; ``` *Output* ```none Text theme={null} make_matrix_2x3((2), (3), (1), (2), (3), (4), (5), (6)) -------------------------------------------------------------------------------- [[1.0,2.0,3.0],[4.0,5.0,6.0]] Fetched 1 row ``` Create a row vector with four doubles. ```sql SQL theme={null} SELECT _R{1,2,3,4} FROM sys.dummy1; ``` *Output* ```none Text theme={null} _r{1,2,3,4} -------------------------------------------------------------------------------- [[1.0,2.0,3.0,4.0]] Fetched 1 row ``` For more examples, see [Matrix Functions and Operators](/matrix-functions-and-operators). ## Related Links [Array Functions and Operators](/array-functions-and-operators) [Tuple Functions and Operators](/tuple-functions-and-operators) [Matrix Functions and Operators](/matrix-functions-and-operators) [Generate Tables Using sys.dummy](/generate-tables-using-sys-dummy) # Attribute Functions Source: https://docs.ocient.com/attribute-functions Reference for OcientGeo attribute functions used to inspect, extract, and modify properties of geospatial geometries such as points, linestrings, and polygons. attribute functions return descriptive information on the specified data set. ## ST\_COORDDIM Alias for ST\_NDIMS or ST\_NDIMENSION. Returns an `INTEGER` of the coordinate dimension of the specified geography. **Syntax** ```sql SQL theme={null} ST_COORDDIM(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------------------------- | ------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, or `POLYGON` | The geospatial object used to calculate the dimensions. | **Example** ```sql SQL theme={null} SELECT ST_COORDDIM(ST_POLYGON('POLYGON((1 1, 2 1, 2 2, 1 1))')); ``` *Output*: `2` ## ST\_DIMENSION Returns an `INTEGER` that represents the dimension of the specified geography. This table describes the result values of the `ST_DIMENSION` function. | **ST\_DIMENSION Result Value** | **Dimension Type** | | ------------------------------ | --------------------- | | `-1` | Empty geography value | | `0` | `POINT` | | `1` | `LINESTRING` | | `2` | `POLYGON` | The `ST_DIMENSION` function always returns `0` for non-empty `POINT` objects. For non-empty `LINESTRING` and `POLYGON` objects, the `ST_DIMENSION` function returns the dimension value equal to the greatest dimension represented by their bounding points. For example, a `LINESTRING` or `POLYGON` object that contains only a single point returns `0`. Similarly, a `POLYGON` object can return `1` if it contains insufficient points to represent a closed polygon. **Syntax** ```sql SQL theme={null} ST_DIMENSION(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------------------------- | ------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, or `POLYGON` | The geospatial object used to calculate the dimensions. | **Examples** In this example, the function evaluates a `LINESTRING` with an empty `POINT` value. ```sql SQL theme={null} SELECT ST_DIMENSION(ST_LINESTRING('POINT EMPTY')); ``` *Output*: `-1` In this example, the function evaluates a `LINESTRING` with a single `POINT` value. ```sql SQL theme={null} SELECT ST_DIMENSION(ST_LINESTRING('ST_POINT(0 0)')); ``` *Output*: `0` This example evaluates an empty `LINESTRING`. ```sql SQL theme={null} SELECT ST_DIMENSION(ST_LINESTRING('LINESTRING EMPTY')); ``` *Output*: `-1` In this example, the function again evaluates a `LINESTRING` with a single `POINT` value. ```sql SQL theme={null} SELECT ST_DIMENSION(ST_LINESTRING('LINESTRING(0 0)')); ``` *Output*: `0` In this example, the function evaluates a `LINESTRING` with multiple `POINT` values. ```sql SQL theme={null} SELECT ST_DIMENSION(ST_LINESTRING('LINESTRING(0 0, 1 1)')); ``` *Output*: `1` This example evaluates an empty `POLYGON`. ```sql SQL theme={null} SELECT ST_DIMENSION(ST_POLYGON('POLYGON EMPTY')); ``` *Output*: `-1` This example evaluates a `POLYGON` with a single `POINT`. ```sql SQL theme={null} SELECT ST_DIMENSION(ST_POLYGON('POLYGON((1 1))')); ``` *Output*: `0` This example evaluates a `POLYGON` with two `POINT` values. ```sql SQL theme={null} SELECT ST_DIMENSION(ST_POLYGON('POLYGON((0 0, 1 1))')); ``` *Output*: `1` This example evaluates a `POLYGON` with multiple `POINT` values, enough to close the `POLYGON`. ```sql SQL theme={null} SELECT ST_DIMENSION(ST_POLYGON('POLYGON((0 0, 1 0, 1 1, 0 0))')); ``` *Output*: `2` ## ST\_GEOMETRYTYPE Returns a `STRING` that represents the geometry of the input geospatial value. Supported values are `ST_POINT`, `ST_LINESTRING`, and `ST_POLYGON`. This function enforces the same strict dimension types as the [ST\_DIMENSION](#st_dimension) function. If `geo` is NULL, then the function returns NULL. **Syntax** ```sql SQL theme={null} ST_GEOMETRYTYPE(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------- | ---------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, `POLYGON` | A geospatial value to be identified by its dimension type. | **Example** ```sql SQL theme={null} SELECT ST_GEOMETRYTYPE(ST_POINT(1, 2)); ``` *Output*: `ST_POINT` ## ST\_ISEMPTY Returns `TRUE` if the specified geography value is empty, such as `'POLYGON EMPTY'`. A NULL input value returns NULL. **Syntax** ```sql SQL theme={null} ST_ISEMPTY(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------- | ------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, `POLYGON` | A geospatial value to be examined if it is empty. | **Example** ```sql SQL theme={null} SELECT ST_ISEMPTY(ST_POLYGON('POLYGON EMPTY')); ``` *Output*: `TRUE` ## ST\_MEMSIZE Returns an `INTEGER` representing the number of bytes in memory required to store the specified geography. **Syntax** ```sql SQL theme={null} ST_MEMSIZE(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------- | ----------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, `POLYGON` | A geospatial value to be measured by its data memory. | **Example** ```sql SQL theme={null} SELECT ST_MEMSIZE(ST_POINT(1,1)); ``` *Output*: `16` ## ST\_NDIMS or ST\_NDIMENSION Alias for [ST\_COORDDIM](#st_coorddim). ## ST\_NPOINTS or ST\_NUMPOINTS Returns an `INTEGER` representing the number of `POINT` values in a specified geography. If the specified value is a `POLYGON`, this function counts the number of `POINT` values in both the exterior and any holes. **Syntax** ```sql SQL theme={null} ST_NPOINTS(geo) ``` ```sql SQL theme={null} ST_NUMPOINTS(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------- | ------------------------------------------------------------------ | | `geo` | `POINT`, `LINESTRING`, `POLYGON` | A geospatial value to be examined for its count of `POINT` values. | **Example** ```sql SQL theme={null} SELECT ST_NPOINTS( ST_POLYGON( 'POLYGON((1 2, 1 3, 1 2))')); ``` *Output*: `3` ## ST\_SRID Returns the EPSG code of the spatial reference identifier (SRID) of the input geography. All geographies are of type GCS WGS 84, which is identified by the value 4326. There is no way to set a different SRID. **Syntax** ```sql SQL theme={null} ST_SRID(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------- | ---------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, `POLYGON` | A geospatial value to be analyzed for its EPSG code. | **Example** ```sql SQL theme={null} SELECT ST_SRID(ST_MAKEPOINT(1.1, 3.11)); ``` *Output*: `4326` ## ST\_X Returns the x value, or longitude, of the specified `POINT`. The returned value is a `DOUBLE` type. **Syntax** ```sql SQL theme={null} ST_X(point) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ----------------------------------------------------------------- | | `point` | `POINT` | A geospatial point to be analyzed for its x value, the longitude. | **Example** ```sql SQL theme={null} SELECT ST_X(ST_POINT(3, 5)); ``` *Output*: `3` ## ST\_XMAX Returns the maximum x value of the specified geography. The returned value is a `DOUBLE` type. **Syntax** ```sql SQL theme={null} ST_XMAX(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------- | --------------------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, `POLYGON` | One or more geospatial points to be analyzed for the maximum x value. | **Example** In this example, the `ST_XMAX` function returns `2` because it is the largest x value of the specified points. ```sql SQL theme={null} SELECT ST_XMAX( ST_POLYGON(ST_LINESTRING('LINESTRING(2 5, 1 2, 1 4, 0 3)'))); ``` *Output*: `2` ## ST\_XMIN Returns the minimum x value of the specified geography. **Syntax** ```sql SQL theme={null} ST_XMIN(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------- | --------------------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, `POLYGON` | One or more geospatial points to be analyzed for the minimum x value. | **Example** In this example, the `ST_XMIN` function returns `0` because it is the smallest x value of the specified points. ```sql SQL theme={null} SELECT ST_XMIN( ST_POLYGON(ST_LINESTRING('LINESTRING(2 5, 1 2, 1 4, 0 3)'))); ``` *Output*: `0` ## ST\_Y Returns the y value, or latitude, of the specified `POINT`. The returned value is a `DOUBLE` type. **Syntax** ```sql SQL theme={null} ST_Y(point) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ---------------------------------------------------------------- | | `point` | `POINT` | A geospatial point to be analyzed for its y value, the latitude. | **Example** ```sql SQL theme={null} SELECT ST_Y(ST_POINT(3, 5)); ``` *Output*: `5` ## ST\_YMAX Returns the maximum y value of the specified geography. **Syntax** ```sql SQL theme={null} ST_YMAX(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------- | --------------------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, `POLYGON` | One or more geospatial points to be analyzed for the maximum y value. | **Example** In this example, the `ST_YMAX` function returns `5` because it is the largest y value of the specified points. ```sql SQL theme={null} SELECT ST_YMAX( ST_POLYGON(ST_LINESTRING('LINESTRING(2 5, 1 2, 1 4, 0 3)'))); ``` *Output*: `5` ## ST\_YMIN Returns the minimum y value of the specified geography. **Syntax** ```sql SQL theme={null} ST_YMIN(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------- | --------------------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, `POLYGON` | One or more geospatial points to be analyzed for the minimum y value. | **Example** In this example, the `ST_YMIN` function returns `2` because it is the smallest y value of the specified points. ```sql SQL theme={null} SELECT ST_YMIN( ST_POLYGON(ST_LINESTRING('LINESTRING(2 5, 1 2, 1 4, 0 3)'))); ``` *Output*: `2` ## Related Links [Geospatial Data Types](/data-types#geospatial-data-types) [Linestring Constructors](/linestring-constructors) [Point Constructors](/point-constructors) [Polygon Constructors](/polygon-constructors) # Authentication Methods Source: https://docs.ocient.com/authentication-methods Overview of authentication methods supported by Ocient, including database password, single sign-on (SSO), OpenID Connect, and token-based access. offers two authentication methods to access the database: Password authentication and Single Sign On (SSO) authentication. You can use these independently or as an organization according to your security and access control needs. This section explains each authentication method and how it can be configured on Ocient. ## Password-Based Authentication Users created using DCL are required to set a password that the system uses for authentication when connecting to the database. Administrators can use [Database Password Security Settings](/database-password-security-settings) to manage settings such as minimum password length and password expiration policies. ### Fully Qualified User Name (FQUN) To connect to the database, you must provide an FQUN. For database users, the FQUN has this form. ```Text Text theme={null} @ ``` For example, the FQUN of the user `alice`, a member of the database `example_database`. ```Text Text theme={null} alice@example_database ``` ### How To Connect using Password Authentication Set the FQUN in the connection string along with the password of the user. For example, this code is the command `alice` uses to connect to `example_database` using the Ocient JDBC driver. ```Text Text theme={null} connect to jdbc:ocient://:/example_database;user=alice@example_database;password=****; ``` The following Ocient driver properties must be set when connecting using password authentication: * user — `alice@example_database` * password — \`\` ## Single Sign On (SSO) The Ocient System allows administrators to add a Single Sign-On integration, allowing users to authenticate using an external Identity Provider (idP). A database, including the `system` database, can have 0 or 1 SSO integrations. The existence of an SSO integration has no effect on users who authenticate with the Password Authentication flow. **Supported Protocols:** The Ocient System supports the following SSO protocols: | **Protocol** | **Description** | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [OpenID Connect](https://openid.net/) | OpenID Connect (OIDC) is a simple JSON-/REST-based identity protocol built on top of the OAuth 2.0 protocol. It enables client applications to rely on the authentication performed by an OpenID Connect Provider to verify the identity of a user. Clients can also obtain basic profile information about a user in an interoperable and REST-like manner from OpenID Connect Providers. | ## OpenID Connect There are two flows that allow users to connect to the database using an external OpenID Connect Provider. ### Authorization Code Flow This is the most commonly used authentication method and makes use of the [OAuth 2.0 Authorization Code Grant](https://www.oauth.com/oauth2-servers/server-side-apps/authorization-code/). When you connect to the database, the Ocient driver directs you to the authorization endpoint of the OpenID Provider, where you log in with your SSO credentials. After the Provider authenticates you, it redirects you back to the database with a temporary code, which the database exchanges for an ID token, Access token, and optionally, a Refresh token. See this image for a sequence diagram of the flow. Sequence of authentication flow **Connect using the Authorization Code Flow** The client handshake method must be set to SSO, and the connection username and password set to the empty string. For example, this code is the command any user would use to connect to the `example_database` database using the Ocient JDBC driver. ```Text Text theme={null} connect to jdbc:ocient://:/example_database;handshake=SSO;user=;password=; ``` These Ocient driver properties must be set when connecting using the Authorization Code flow: * handshake — `SSO` * user — `` * password — `` The `user` and `password` properties are required and must be set to the empty string. ### SSO Token Flow Users can provide either an [ID Token](https://openid.net/specs/openid-connect-core-1_0.html#IDToken) or an [Access Token](https://datatracker.ietf.org/doc/html/rfc6749#section-1.4) issued by the OpenID Provider to connect to the database. The provided ID or Access token must contain these claims: * iss — The issuer must match the issuer contained in the discovery document of the OpenID Provider. * aud — The token audience must contain the `client_id` used when configuring the OpenID Provider in the database. Ocient recommends including the `email` scope in any token provided to Ocient. The system uses the value of the email claim to identify the user in the database audit trails. Ocient supports these signing algorithms for ID Tokens: * RSA 256 * RSA 384 * RSA 512 **Connect using the SSO Token Flow** These Ocient driver properties must be set when connecting using an OpenID token: * handshake — `SSO` * user — `id_token` or `access_token` * password — `` The handshake must be set to SSO, with the username set to the token type and the password set to the token payload. For example, this format is the JDBC command to connect to the `example_database` database using an ID Token. ```Text Text theme={null} connect to jdbc:ocient://:/example_database;handshake=SSO;user=id_token;password=; ``` This format is the JDBC command to connect to `example_database` database using an Access Token. ```Text Text theme={null} connect to jdbc:ocient://:/example_database;handshake=SSO;user=access_token;password=; ``` ### Configure the OpenID Provider Before users can authenticate using SSO, you must register the database with the OpenID Provider. This process varies depending on the provider, but typically, the steps for doing so are: 1. Create an application for the Ocient System in the Provider. Select the "Native Application" application type. 2. Enable the Authorization Code, Refresh Token, and Device Authorization grant types. 3. Grant permission to the appropriate users to use the newly created application. 4. Enter this Redirect URI. | **Application** | **Redirect URI** | | --------------- | ------------------------------------------------- | | JDBC Driver | `http://localhost:7050/ocient/oauth2/v1/callback` | | OpenAPI or UI | `https:///v1/callback` | For the OpenAPI or UI application, the redirect URI must use `https` format. The connectivity pool defines the OpenAPI or WebUI `advertized_ip` setting. For more information, see [CONNECTIVITY POOL](/cluster-and-node-management#connectivity-pool). Record the token `issuer` and `client_id` of the newly created application. You need these values to configure the Ocient System. ### SSO Parameters These SSO parameters are configurable by using DDL statements. | **Name** | **Required** | **Value Type** | **Default Value** | **Description** | | -------------------------------- | ------------ | ------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `issuer` | Yes | string | None | The [complete URL](https://openid.net/specs/openid-connect-core-1_0.html#Terminology) for the OAuth 2.0 and OpenID Connect Authorization Server. This property value is the expected `\"iss\"` claim in access tokens validated by the database. | | `client_id` | Yes | string | None | The [client identifier](https://openid.net/specs/openid-connect-core-1_0.html#Terminology) as registered with the OpenID Provider. | | `client_secret` | Yes | string | None | The [client secret](https://openid.net/specs/openid-connect-core-1_0.html#IDTokenValidation) as registered with the OpenID Provider. This property is required for some SSO workflows. | | `default_group` | Yes | string | None | The group users are assigned if no group is specified in the `group_claim_mappings` property for the OpenID Provider. | | `redirect_host` | No | string | Depends on the connector (pyocient or JDBC). | Specifies the host name during SSO redirection. | | `redirect_ssl` | No | Boolean | Depends on the connector (pyocient or JDBC). | Set to `true` to enable SSL callback during SSO redirection (i.e., redirect uses `https`).
Set to `false` to disable (i.e., redirect uses `http`). | | `disabled` | No | Boolean | `false` | Set to `true` to disable the OIDC integration for maintenance temporarily.
⚠️ If you set this property to `true`, all authentication requests using this connection fail. | | `enable_id_token_authentication` | No | Boolean | `false` | Set this property to `true` if the identifier token also contains the authorization token.
In most circumstances, this option is necessary only for machine-to-machine connections without user interaction, such as a server using a script to connect to an Ocient System. | | `user_id_claims` | No | list of strings (e.g., value \[, ...] | \["email"] | Set the identifier token claims used to identify users in audit trails. If you do not set a value, the system uses the `"email"` claim if it is present, otherwise it uses `["iss", "sub"]`. | | `additional_scopes` | No | list of strings (e.g., value \[, ...] | \[] | Specifies additional scopes to request when executing the Authorization Code Flow. | | `additional_audiences` | No | list of strings (e.g., value \[, ...] | \[] | Specifies additional audiences to accept when validating tokens. This property is useful for authorization servers without the token exchange capability.
Each additional audience is a case-sensitive URL from a provider, similar to the `issuer` value. | | `groups_claim_ids` | No | list of strings (e.g., value \[, ...] | \[] | Specifies the token claims that can be used to map the user to a Database group. If you specify the `groups_claim_mappings` property, you must also specify the `groups_claim_ids` property. | | `groups_claim_mappings` | No | map of strings (e.g., key = value \[, ...]) | Specifies mappings from the Provider group to the Database group. | | | `roles_claim_ids` | No | list of strings (e.g., value \[, ...] | \[] | Specifies the token claims that can be used to map the user to a Database role. If you specify the `roles_claim_mappings` property, you must also specify the `roles_claim_ids` property. | | `roles_claim_mappings` | No | map of strings (e.g., key = value \[, ...]) | Specifies mappings from the Provider role to the Database role. | | | `allowed_groups` | No | list of strings (e.g., value \[, ...] | \[] | Specifies a list of external identity provider groups that are permitted to authenticate through this SSO integration. If you specify the `allowed_groups` property, only users who are members of the specified identify provider groups can access the Ocient System.

If this property is empty or you do not specify it, then group-based filtering is disabled, and the system allows all authenticated users from the provider unless they are explicitly blocked. Group names must match exactly as they appear in the claims of the identify provider group. | | `allowed_roles` | No | list of strings (e.g., value \[, ...] | \[] | Defines a list of external identity provider roles that are authorized to access the Ocient System through this SSO integration. Only users assigned to the specified roles in the external provider can authenticate.

If this property is empty or you do not specify it, role-based access control is disabled for this integration. Role names must correspond exactly to the role claims provided by the identity provider in the authentication response. | | `blocked_groups` | No | list of strings (e.g., value \[, ...] | \[] | Specifies a list of external identity provider groups that are explicitly denied access through this SSO integration. The system blocks users who are members of any specified group in the external identity provider from authenticating, regardless of other permissions they have. Setting this property takes precedence over the `allowed_groups` property if a user belongs to both an allowed group and a blocked group. Group names must match exactly as they appear in the claims of the identify provider group. | | `blocked_roles` | No | list of strings (e.g., value \[, ...] | \[] | Defines a list of external identity provider roles that are explicitly prohibited from accessing the Ocient System through this SSO integration. The system denies access to users assigned to any of the specified roles, overriding any other access permissions. Setting this property takes precedence over the
`allowed_roles` property in cases where a user is in both an allowed role and a blocked role. Role names must correspond exactly to the role claims of the identity provider group. | | `allow_offline_access` | No | Boolean | `false` | When the `allow_offline_access` property is `true`, Ocient requests offline access from the OpenID Connect identity provider by including `access_type=offline` and `prompt=consent` in the authorization URL. | After all parameters have been retrieved, a database administrator can execute a DDL SQL statement using this syntax. ```sql SQL theme={null} theme={null} ALTER DATABASE database { SET | ALTER } SSO INTEGRATION sso_protocol [, ...] ::= -- literal or string property_name = value | -- list property_name = [ value [, ...] ] | -- map property_name = { key = value [, ...] } # map ``` ```sql SQL theme={null} theme={null} ALTER DATABASE example_database SET SSO INTEGRATION oidc issuer = "https://example.com", client_id = "xxxxxx", default_group = "default", additional_scopes = [email, groups, profile, offline_access], user_id_claims = [claim1, claim2], groups_claim_ids = [claim3], groups_claim_mappings = { idp_group1 = ocient_group1, idp_group2 = ocient_group2 }; ``` * The user must be a database administrator (or have the "Security Administrator" role for the "system" database). * The user must be connected to the database. * Any String key or value can be placed between double quotations (e.g., `default_group` OR `"default_group"`). * String values that contain characters other than `([a-zA-z] | [0-9] | '_')` must be placed between double quotations (e.g., `this.is.a.complex-$tring` ⇒ `"this.is.a.complex-$tring"`). * The `NULL` value can be used to clear existing configurations of `LIST` or `MAP` properties (e.g., `user_claim_ids = NULL`). To remove the connection, execute this SQL statement. ```sql SQL theme={null} ALTER DATABASE example_database REMOVE SSO INTEGRATION; ``` The OpenID Connect protocol requires TLS for communication between the end user, the database, and the OpenID Provider. A valid TLS key and certificate pair must be configured on the SQL Nodes, which behave as a server during the Authorization Code Flow. ### Set Up Machine-To-Machine SSO Integration You can use DDL SQL statements to add client credentials for SSO connection to individual databases or to a connectivity pool that operates across SQL Nodes. These steps set up SSO client credentials on a database, but can also apply to connectivity pools. **Create an SSO protocol.** Use the [CREATE SSO INTEGRATION](/cluster-and-node-management#create-sso-integration) SQL statement to make an SSO integration protocol with your preferred configuration. This configuration must include these properties: * The `enable_id_token_authentication` SSO property must be set to `true`. * The `additional_audiences` SSO property must use the URL from the OIDC provider, similar to the `issuer` value. ```sql SQL theme={null} CREATE SSO INTEGRATION sso_test PROTOCOL oidc issuer = "https://accounts.google.com", client_id = example_database_app_id, default_group = example_database_group, enable_id_token_authentication = true, additional_audiences = ["https://accounts.google.com"]; ``` **Assign the SSO protocol.** Use the [ALTER DATABASE SET SSO INTEGRATION](/databases#alter-database-set-sso-integration) SQL statement to integrate the SSO protocol into a database. For connectivity pools, use the [ALTER CONNECTIVITY\_POOL SET SSO INTEGRATION](/cluster-and-node-management#alter-connectivity_pool-set-sso-integration) SQL statement. ```sql SQL theme={null} ALTER DATABASE example_database SET SSO INTEGRATION sso_test; ``` **Connect to your system.** Use a JDBC connection string that includes the access token value ``. This string connects to the local host at port number `4050`, uses the SSO protocol, and with the username `id_token`. ```shell Shell theme={null} CONNECT TO jdbc:ocient://localhost:4050/system;handshake=SSO;user=id_token;password= ``` ### Set Up Cross-Database SSO Integration Set up authentication using SSO integration across databases. Identify the database that has SSO integration using the `username@database` format, where `username` is the access or identifier token and `database` is the database for the SSO integration. Each SSO integration has a client identifier `client_id`. You must obtain the access token using the same client identifier used in the SSO integration for authentication. When you authenticate using SSO integration as a user, the system assigns you to a group defined in the integration. Such users always have groups qualified with the SSO integration of the database. These groups have fully qualified names such as `sso_users@system`, where `sso_users` is the name of the group and `system` is the name of the database. You can use a fully qualified group name from any database. Without the `@database` qualification, the referenced group specifies the current database. To grant cross-database privileges to such groups, you must reference them using their fully qualified names. You can find a list of the groups the current user is a member of by using the [CURRENT\_GROUPS](/other-functions-and-expressions#current_groups) function. To manage privileges and groups, use the `GRANT`, `REVOKE`, `ALTER GROUP`, and `DROP GROUP` SQL statements. For details about privileges, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). **Requirements** * Configure the SSO integration in the source database. * The SSO user must be a member of a group that has the USE privilege on the target database. If the SSO user is not a member of any groups, the default group for the SSO integration must have the USE privilege on the target database. * The token (either an access or an identifier token) must be valid for the SSO integration in the source database. **Create a Cross-Database SSO Integration** These steps guide you through setting up a cross-database SSO integration. Create a default group `sso_users`. ```sql SQL theme={null} CREATE GROUP sso_users; ``` Create the SSO integration `test_sso` in the source database using the `CREATE SSO INTEGRATION` SQL statement. Specify your issuer and client identifier for the `issuer` and `client_id` parameters. ```sql SQL theme={null} CREATE SSO INTEGRATION test_sso PROTOCOL OIDC enable_id_token_authentication = true, issuer = 'https://your-idp.com/oidc', client_id = 'your_client_id', default_group = sso_users, additional_scopes = [profile, email, groups, offline_access], protocol_version = 'v1', groups_claim_ids = [groups]; ``` Set the SSO integration as the default for the database `source_db`. This step is optional. To set up SSO integration for the `system` database, substitute `source_db` with `system`. ```sql SQL theme={null} ALTER DATABASE source_db SET SSO INTEGRATION test_sso; ``` Grant cross-database privileges to the target database `target_db` for the default group `sso_users`. ```sql SQL theme={null} GRANT USE ON DATABASE target_db TO GROUP sso_users; ``` Connect to the target database `target_db` at the local host with port number `4050` using the JDBC driver with an access token on the source database `access_token@source_db`. Specify a password token. ```none Text theme={null} Properties props = new Properties(); props.setProperty("user", "access_token@source_db"); props.setProperty("password", token); props.setProperty("handshake", "sso"); Connection conn = DriverManager.getConnection("jdbc:ocient://host:4050/target_db", props); ``` Or, connect using the device flow by using the name of the source database `@source_db`. ```none Text theme={null} Properties props = new Properties(); props.setProperty("user", "@source_db"); props.setProperty("handshake", "sso"); props.setProperty("ssoOAuthFlow", "deviceGrant"); Connection conn = DriverManager.getConnection("jdbc:ocient://host:4050/target_db", props); ``` Verify your connection using these queries: Check the current user using the [CURRENT\_USER](/other-functions-and-expressions#current_user) function. ```sql SQL theme={null} SELECT CURRENT_USER(); ``` Check the current groups using the CURRENT\_GROUPS function. ```sql SQL theme={null} SELECT CURRENT_GROUPS(); ``` Check the current database using the [CURRENT\_DATABASE](/other-functions-and-expressions#current_database) function. ```sql SQL theme={null} SELECT CURRENT_DATABASE(); ``` View session information using the `sys.sessions` system catalog table and the identifier of the current session by using the [CURRENT\_SESSION\_ID](/other-functions-and-expressions#current_session_id) function. ```sql SQL theme={null} SELECT * FROM sys.sessions WHERE id = CURRENT_SESSION_ID(); ``` You can also retrieve information about the defined groups using the `sys.groups` system catalog table. **Driver Connection Properties** These tables describe the connection properties you can specify. **pyocient Parameters** This table lists the DNS parameters that apply to SSO integration. For details, see [Ocient Python Module: pyocient](/ocient-python-module-pyocient). | **Parameter** | **Description** | **Example** | | ------------------ | -------------------------------------------- | ------------------------------------------ | | `handshake` | Authentication method to use for connection. | `sso` | | `ssoOAuthFlow` | Type of OAuth flow. | `deviceGrant`
or `authenticationCode` | | `identityprovider` | The name of the SSO integration. | `test_sso` | **JDBC Driver Properties** This table lists the JDBC connection properties that apply to SSO integration. For details, see [CONNECT](/commands-supported-by-the-ocient-jdbc-cli-program#connect). | **Property** | **Description** | **Example** | | ------------------ | -------------------------------------------- | ------------------------------------------ | | `handshake` | Authentication method to use for connection. | `sso` | | `ssoOAuthFlow` | Type of OAuth flow. | `deviceGrant`
or `authenticationCode` | | `identityprovider` | The name of the SSO integration. | `test_sso` | ### User Access Control and Workload Management with SSO The database maintains an internal User Access Control model consisting of groups, roles, and service classes. The database relies on group or role membership to determine the privileges of a user or service class. The following integration properties allow administrators to grant SSO users membership to a Database group or role: * The `groups_claim_ids` and `roles_claim_ids` properties define the token claims that specify the Provider-defined groups or roles of the user. * The `groups_claim_mappings` and `roles_claim_mappings` properties define mappings between a Provider and Database groups or roles. * The `additional_scopes` property should include any request scopes needed for the Provider to include the group or role claims in tokens it issues. Because SSO users can map to multiple groups, when choosing which group to assign creator privileges, the system follows this criteria: * Filter by authorization using only groups with CREATE privileges on the relevant schema or database. * When multiple authorized groups exist at different levels, the system uses hierarchy precedence to ensure that privileges are assigned at the most specific scope. The enforced hierarchy is that schema-level group privileges take precedence over database-level group privileges. * If multiple groups match at the same level, the system selects the group name by using alphabetical order. For example, if an SSO user belongs to the `analytics_team` group (with CREATE privileges at the database level) and `finance_schema_admins` group (with CREATE privileges at the schema level), and creates a table in the `finance` schema, the system assigns creator privileges to `finance_schema_admins`. For details about privileges, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). Consider an example where the Provider lists all groups the user belongs to in the token claim `"user_groups"` when the application requests the `"example_scope"` scope. In this scenario, the administrator wants to associate the Provider Group `"Example Provider Group"` with the Database Role `"example_database Analyst"`. ```sql SQL theme={null} ALTER DATABASE example_database ALTER SSO INTEGRATION oidc /* Tell the Database to request the "profile", "email" and "offline_access" scopes */ additional_scopes = [ profile, email, offline_access ], /* Tell the Database to look for the "user_groups" token claim to establish role membership */ roles_claim_ids = [ user_groups ], /* Map provider group "Example Provider Group" to database role "example_database Analyst" */ roles_claim_mappings = { "Example Provider Group" = "example_database Analyst" }; ``` By default, the database grants all users membership to the "Public Role" of the database for the current connection. Unlike other default roles, you can grant additional privileges to the "Public Role". Additionally, database administrators can revoke access to users with membership to any of the specified Provider groups or roles. To revoke access, execute this statement. ```sql SQL theme={null} ALTER DATABASE example_database ALTER SSO INTEGRATION oidc blocked_groups = [ [, ...] ], blocked_roles = [ [, ...] ]; ``` The user must have the ALTER privilege on all Ocient-defined groups or roles specified by `groups_claim_mappings`, `roles_claim_mappings`, `blocked_groups`, and `blocked_roles`. ### User Claim Identifiers (IDs) By default, audit trails, including but not limited to system-level log messages, identify users connecting using SSO by the `email` claim of the ID Token. If the `email` claim is not present in the token, the identifier is `iss` and `sub` ID Token claims separated by `::`, for example: `"https://ocient.okta.com::00u5rslndgXm9Ey7y5d7"`. To change the FQUN, alter the `user_claim_ids` SSO integration property. For example, to change the FQUN to a combination of the `first_name` and `last_name` claim values, execute this SQL statement. ```sql SQL theme={null} ALTER DATABASE example_database ALTER SSO INTEGRATION oidc user_claim_ids = [first_name, last_name, email]; ``` After executing this statement, users are identified with the `::::` format. (For example: `John::Smith::jsmith@example.com`) Updating the `user_claim_ids` property has no effect on users that are already connected to the database. ### System Catalog Tables Properties that do not contain sensitive data can be viewed using the `sys.oidc_integrations` virtual table. In addition to the configurable properties, a `database_id` column is included in the schema. To view the configuration for a specific database (if one exists), execute this SQL statement. ```sql SQL theme={null} SELECT * FROM sys.databases JOIN sys.oidc_integrations ON databases.id = oidc_integrations.database_id WHERE databases.name = 'example_database'; ``` The user must be a database administrator to view the OpenID Connect properties. ## Related Links [Data Definition Language (DDL) Statement Reference](/data-definition-language-ddl-statement-reference) [SQL Reference](/sql-reference) [System Catalog](/system-catalog) # Backup and Restore Source: https://docs.ocient.com/backup-and-restore Back up and restore Ocient databases using the BACKUP and RESTORE SQL statements to protect data, recover from failures, and migrate between environments. can back up and restore a single node or a multi-node system. The backup and restore procedures can replace a node in an existing Ocient cluster due to an OS drive failure or hardware upgrade. Exact backup and restore procedures depend on the disk layout of your Ocient nodes. Consult Ocient Support or the system administrator who installed your Ocient cluster to determine the best backup strategy for your deployment. ## Related Links [Maintenance Overview](/maintenance-overview) # Categorized SQL Functions List Source: https://docs.ocient.com/categorized-sql-functions-list Categorized reference of Ocient SQL functions grouped by purpose, including aggregate, conversion, date and time, math, string, spatial, and machine learning. | **Function Name** | **Category Name** | **Function Description** | | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [ACCURACY\_SCORE](/aggregate-functions#accuracy_score) | Aggregate Functions | Returns the fraction of predictions that match the actual class labels. | | [ANY\_VALUE](/aggregate-functions#any_value) | Aggregate Functions | Returns an arbitrary, non-NULL value from the input column. | | [APPROX\_COUNT\_DISTINCT](/aggregate-functions#approx_count_distinct) | Aggregate Functions | Approximate distinct count by using hyper-log-log (95% confidence interval that the value is within 4.5%). | | [APPROX\_SUM](/aggregate-functions#approx_sum) | Aggregate Functions | Allows the aggregation engine to use a faster, non-deterministic ordering to summate floating-point columns. | | [ARRAY\_AGG](/aggregate-functions#array_agg) | Aggregate Functions | Returns an array containing every row from the expression. | | [ARRAY\_CONCAT\_AGG](/aggregate-functions#array_concat_agg) | Aggregate Functions | Returns an array that concatenates arrays across rows. | | [AVG](/aggregate-functions#avg) | Aggregate Functions | Average, or arithmetic mean, over the set. | | [COEFFICIENT\_OF\_DETERMINATION](/aggregate-functions#coefficient_of_determination) | Aggregate Functions | Computes the coefficient of determination (R²) between actual and predicted values. | | [CONFUSION\_MATRIX](/aggregate-functions#confusion_matrix) | Aggregate Functions | Returns a structured representation of the counts for every combination of actual and predicted class labels relative to a specified positive class. | | [CORR](/aggregate-functions#corr) | Aggregate Functions | Alias for CORRELATION. | | [CORRELATION](/aggregate-functions#correlation) | Aggregate Functions | Sample correlation. | | [CORRELATIONP](/aggregate-functions#correlationp) | Aggregate Functions | Population correlation. | | [COUNT](/aggregate-functions#count) | Aggregate Functions | Number of rows in the set. | | [COVAR\_POP](/aggregate-functions#covar_pop) | Aggregate Functions | Alias for COVARIANCEP. | | [COVAR\_SAMP](/aggregate-functions#covar_samp) | Aggregate Functions | Alias for COVARIANCE. | | [COVARIANCE](/aggregate-functions#covariance) | Aggregate Functions | Sample covariance. | | [COVARIANCEP](/aggregate-functions#covariancep) | Aggregate Functions | Population covariance. | | [F1\_SCORE](/aggregate-functions#f1_score) | Aggregate Functions | Returns the harmonic mean of precision and recall for a specified positive class. | | [KURTOSIS](/aggregate-functions#kurtosis) | Aggregate Functions | The sample over the set. | | [KURTOSISP](/aggregate-functions#kurtosisp) | Aggregate Functions | The population over the set. | | [MAX](/aggregate-functions#max) | Aggregate Functions | Maximum value in the specified column. | | [MEAN\_ABSOLUTE\_ERROR](/aggregate-functions#mean_absolute_error) | Aggregate Functions | Returns the mean absolute error (MAE) between actual and predicted values. | | [MEAN\_ABSOLUTE\_PERCENTAGE\_ERROR](/aggregate-functions#mean_absolute_percentage_error) | Aggregate Functions | Returns the mean absolute percentage error (MAPE) between actual and predicted values. | | [MIN](/aggregate-functions#min) | Aggregate Functions | Minimum value in the specified column. | | [PRECISION\_SCORE](/aggregate-functions#precision_score) | Aggregate Functions | Returns the precision score for a specified positive class. | | [PRODUCT](/aggregate-functions#product) | Aggregate Functions | Product over the set. | | [RECALL\_SCORE](/aggregate-functions#recall_score) | Aggregate Functions | Returns the recall score for a specified positive class. | | [ROC\_AUC\_SCORE](/aggregate-functions#roc_auc_score) | Aggregate Functions | Returns the area under the receiver operating characteristic (ROC) curve (AUC). | | [SKEW](/aggregate-functions#skew) | Aggregate Functions | The sample skewness over the set of values. | | [SKEWP](/aggregate-functions#skewp) | Aggregate Functions | Computes the population skewness over the set of values. | | [STDEV](/aggregate-functions#stdev) | Aggregate Functions | Sample standard deviation. | | [STDDEV](/aggregate-functions#stddev) | Aggregate Functions | Alias for STDEV. | | [STDDEV\_POP](/aggregate-functions#stddev_pop) | Aggregate Functions | Alias for STDEVP. | | [STDDEV\_SAMP](/aggregate-functions#stddev_samp) | Aggregate Functions | Alias for STDEV. | | [STDEVP](/aggregate-functions#stdevp) | Aggregate Functions | Population standard deviation. | | [STRING\_AGG](/aggregate-functions#string_agg) | Aggregate Functions | Returns a string concatenated from every row from the expression. The delimiter argument is optional. | | [SUM](/aggregate-functions#sum) | Aggregate Functions | Sum over the set. | | [VAR\_POP](/aggregate-functions#var_pop) | Aggregate Functions | Alias for VARIANCEP. | | [VAR\_SAMP](/aggregate-functions#var_samp) | Aggregate Functions | Alias for VARIANCE. | | [VARIANCE](/aggregate-functions#variance) | Aggregate Functions | Sample variance. | | [VARIANCEP](/aggregate-functions#variancep) | Aggregate Functions | Population variance. | | [ARRAY\[\]](/array-functions-and-operators) | Array Functions | -compliant constructor. The type of the array is deduced from the elements. | | [ARRAY\_APPEND](/array-functions-and-operators) | Array Functions | Add value to the back of an array. | | [ARRAY\_ARGMAX](/array-functions-and-operators) | Array Functions | Returns the corresponding argmax of the array as a BIGINT index. | | [ARRAY\_ARGMIN](/array-functions-and-operators) | Array Functions | Returns the corresponding argmin of the array as a BIGINT index. | | [ARRAY\_CAT](/array-functions-and-operators) | Array Functions | Concatenate 2 arrays into a new one. | | [ARRAY\_CAT\_DISTINCT](/array-functions-and-operators#array_cat_distinct) | Array Functions | Concatenates two or more arrays in the order of the input arguments. | | [ARRAY\_DISTINCT](/array-functions-and-operators#array_distinct) | Array Functions | Removes duplicates from an array while preserving the first occurrence order. | | [ARRAY\_LENGTH](/array-functions-and-operators) | Array Functions | Return the number of elements of a given array. | | [ARRAY\_MAX](/array-functions-and-operators) | Array Functions | Returns the corresponding maximum of the array. | | [ARRAY\_MIN](/array-functions-and-operators) | Array Functions | Returns the corresponding minimum of the array. | | [ARRAY\_POSITION](/array-functions-and-operators) | Array Functions | Returns the position of the first matching scalar in the array. | | [ARRAY\_POSITIONS](/array-functions-and-operators) | Array Functions | Returns all elements stored in the right array, and return their respective positions in the left array. | | [ARRAY\_PREPEND](/array-functions-and-operators) | Array Functions | Add value to the front of an array. | | [ARRAY\_REMOVE](/array-functions-and-operators) | Array Functions | Remove value from the array. | | [ARRAY\_REPLACE](/array-functions-and-operators) | Array Functions | Replace value by another in an array. | | [ARRAY\_SUM](/array-functions-and-operators) | Array Functions | Returns the sum of the array. The array must be one-dimensional and contain numeric values. NULL values do not contribute to the sum. | | [ARRAY\_TO\_STRING](/array-functions-and-operators) | Array Functions | Converts array to a list of elements separated by 'delimiter'. | | [CAST\_TO\_ARRAY](/array-functions-and-operators) | Array Functions | Casts the elements of the array to another type. | | [CHAR](/array-functions-and-operators) | Array Functions | Converts array to its string representation. | | [CROSS\_ENTROPY\_LOSS](/array-functions-and-operators) | Array Functions | Returns the cross entropy loss of two arrays. | | [HINGE\_LOSS](/array-functions-and-operators) | Array Functions | Returns the hinge loss of two arrays. | | [LOG\_LOSS](/array-functions-and-operators) | Array Functions | Returns the log loss of two arrays. | | [LOGITS\_LOSS](/array-functions-and-operators) | Array Functions | Returns the logits loss of two arrays. | | [SOFTMAX](/array-functions-and-operators) | Array Functions | Returns the softmax of the array. | | [STRING\_TO\_ARRAY](/array-functions-and-operators) | Array Functions | Converts the string representation of an array (e.g., `'int[1,2,NULL]'`) into an array. | | [TYPE\[\]](/array-functions-and-operators) | Array Functions | Construct an array of SQL type TYPE giving the elements. | | [UNNEST](/array-functions-and-operators#unnest) | Array Functions | Expand each element in an input array out to an individual row. | | [ASCII](/character-and-binary-functions#ascii) | Character and Binary Functions | Returns the ASCII code value of the leftmost character of the character value. | | [BIT\_LENGTH](/character-and-binary-functions#bit_length) | Character and Binary Functions | Returns the length of the character value in bits. | | [BTRIM](/character-and-binary-functions#btrim) | Character and Binary Functions | Alias for TRIM. | | [CHAR\_LENGTH](/character-and-binary-functions#char_length) | Character and Binary Functions | Alias for LENGTH. | | [CHARACTER\_LENGTH](/character-and-binary-functions#character_length) | Character and Binary Functions | Alias for LENGTH. | | [CHR](/character-and-binary-functions#chr) | Character and Binary Functions | Converts an integer value to a string. | | [CONCAT](/character-and-binary-functions#concat) | Character and Binary Functions | Concatenates two values, which must both be either binary, hash, or string data types. This function is equivalent to the \|\| operator. | | [ENDSWITH](/character-and-binary-functions#endswith) | Character and Binary Functions | Returns true if x ends with y and false otherwise. | | [INITCAP](/character-and-binary-functions#initcap) | Character and Binary Functions | For each word in the provided string, capitalize the first character if it is alphabetic. | | [INSTR](/character-and-binary-functions#instr) | Character and Binary Functions | Returns the index position of the first occurrence where the character value char\_substring appears in the character value char by ignoring the case. | | [JSON\_EXTRACT\_PATH\_TEXT](/character-and-binary-functions#json_extract_path_text) | Character and Binary Functions | Returns the value for the key-value pair referenced by a series of path elements in a JSON string. | | [LCASE](/character-and-binary-functions#lcase) | Character and Binary Functions | Alias for LOWER. | | [LEFT](/character-and-binary-functions#left) | Character and Binary Functions | Return the number of characters in the string equal to the value integer. If the integer is negative, the function returns all but the last integer characters. | | [LENGTH](/character-and-binary-functions#length) | Character and Binary Functions | For character data types, this value is in terms of characters. For binary data types, this value is in terms of bytes. | | [LOCATE](/character-and-binary-functions#locate) | Character and Binary Functions | Alias for POSITION. Returns the index position of the first occurrence of the character value substring in character value string. | | [LOWER](/character-and-binary-functions#lower) | Character and Binary Functions | Alias for LCASE. Convert string to lowercase. | | [LPAD](/character-and-binary-functions#lpad) | Character and Binary Functions | Pad the input text to the specified length with the pad string on the left side. | | [LTRIM](/character-and-binary-functions#ltrim) | Character and Binary Functions | Removes leading blanks from the string value string. | | [MD5](/character-and-binary-functions#md5) | Character and Binary Functions | Returns the hexadecimal string (all lowercase) representing the md5 hash of char. | | [MID](/character-and-binary-functions#mid) | Character and Binary Functions | Alias for SUBSTRING. | | [OCTET\_LENGTH](/character-and-binary-functions#octet_length) | Character and Binary Functions | Returns the length in bytes of a character or binary value. | | [POSITION](/character-and-binary-functions#position) | Character and Binary Functions | Alias for LOCATE. | | [REGEXP\_COUNT](/character-and-binary-functions#regexp_count) | Character and Binary Functions | Searches a string for all occurrences of a regular expression pattern. | | [REGEXP\_INSTR](/character-and-binary-functions#regexp_instr) | Character and Binary Functions | Searches a string using a regular expression pattern and returns an integer representing the start position or end position of the substring that matches. | | [REGEXP\_REPLACE](/character-and-binary-functions#regexp_replace) | Character and Binary Functions | Searches a string for all occurrences of a regular expression pattern. | | [REGEXP\_SUBSTR](/character-and-binary-functions#regexp_substr) | Character and Binary Functions | Returns one substring from a string that matches a specified regular expression pattern. | | [REPEAT](/character-and-binary-functions#repeat) | Character and Binary Functions | Repeats the character value char a number of times equal to num. | | [REPLACE](/character-and-binary-functions#replace) | Character and Binary Functions | Replaces all occurrences of substr\_to\_remove in the character value string with substr\_to\_replace. | | [REVERSE](/character-and-binary-functions#reverse) | Character and Binary Functions | Reverse the input string. | | [RIGHT](/character-and-binary-functions#right) | Character and Binary Functions | Return the number of trailing characters in the string equal to the value integer. | | [RPAD](/character-and-binary-functions#rpad) | Character and Binary Functions | Pad the input text to the specified length with the pad string on the right side. | | [RSUBSTRING](/character-and-binary-functions#rsubstring) | Character and Binary Functions | Returns the substring from the right side of a string based on a specified length. | | [RTRIM](/character-and-binary-functions#rtrim) | Character and Binary Functions | Removes leading blanks from the string value string. | | [SHA1](/character-and-binary-functions#sha1) | Character and Binary Functions | Uses the \[SHA-1]\([https://en.wikipedia.org/wiki/SHA-1#:\~:text=In%20cryptography%2C%20SHA%2D1%20(,rendered%20as%2040%20hexadecimal%20digits](https://en.wikipedia.org/wiki/SHA-1#:~:text=In%20cryptography%2C%20SHA%2D1%20\(,rendered%20as%2040%20hexadecimal%20digits).) cryptographic hash function to convert a string into a 40-character string representing the hexadecimal value of a 160-bit checksum. | | [SPACE](/character-and-binary-functions#space) | Character and Binary Functions | Returns a string of repeated spaces equal to the number value, repeat. | | [SPLIT\_PART](/character-and-binary-functions#split_part) | Character and Binary Functions | Split the value string based on the delimiter value. The function returns a substring from the split operation based on the index value (starting from 1). | | [SPLIT\_TO\_ARRAY](/character-and-binary-functions#split_to_array) | Character and Binary Functions | Splits a string into an array of substrings. | | [STARTSWITH](/character-and-binary-functions#startswith) | Character and Binary Functions | Returns true if string starts with substring and false otherwise. | | [STRPOS](/character-and-binary-functions#strpos) | Character and Binary Functions | Equivalent to using LOCATE as LOCATE(substring, string). Note the reversed argument order. | | [SUBSTR](/character-and-binary-functions#substr) | Character and Binary Functions | Alias for SUBSTRING. | | [SUBSTRING](/character-and-binary-functions#substring) | Character and Binary Functions | Returns the substring of a character or binary value. | | [TO\_CHAR](/character-and-binary-functions#to_char) | Character and Binary Functions | Converts a numeric, date, or timestamp value into a CHAR date type. | | [TRANSLATE](/character-and-binary-functions#translate) | Character and Binary Functions | Replaces specified characters in a provided string with a separate set of characters. | | [TRIM](/character-and-binary-functions#trim) | Character and Binary Functions | Alias for BTRIM. Trim leading and trailing blanks from the string. | | [UCASE](/character-and-binary-functions#ucase) | Character and Binary Functions | Alias for UPPER. | | [UPPER](/character-and-binary-functions#upper) | Character and Binary Functions | Convert string to upper case. | | [CASE](/other-functions-and-expressions#case) | Conditional Functions | CASE operates similarly to conditional scripting in other programming languages, allowing it to function like an if / then / else statement or as a switch statement. | | [COALESCE](/other-functions-and-expressions#coalesce) | Conditional Functions | Evaluates to the first argument that is not NULL, or NULL if all arguments are NULL. | | [GREATEST](/other-functions-and-expressions#greatest) | Conditional Functions | Returns the largest non-NULL value of all the arguments, or NULL if all the arguments are NULL. | | [IF\_NULL](/other-functions-and-expressions#if_null) | Conditional Functions | Alias for COALESCE. | | [LEAST](/other-functions-and-expressions#least) | Conditional Functions | Returns the smallest non-NULL value of all arguments, or NULL if all arguments are NULL. | | [MURMUR3](/other-functions-and-expressions#murmur3) | Conditional Functions | [Returns a 32-bit MurmurHash3 hash of the input value as an INTEGER data type.](https://github.com/aappleby/smhasher/blob/master/README.md) | | [NULL\_IF](/other-functions-and-expressions#null_if) | Conditional Functions | Returns the NULL value if two arguments are equal; otherwise, returns the first argument. | | [ZN](/other-functions-and-expressions#zn) | Conditional Functions | If x is NULL, returns 0. Otherwise, returns x. | | [LAG\_VECTORS](/data-preparation#lag_vectors) | Data Preparation | Groups lagged columns generated by the `MULTI_LAGS` or `MULTI_LAGS_ZEROFILL` functions into vector columns. | | [LAGS](/data-preparation#lags) | Data Preparation | Generates a series of lagged columns for a single variable in one statement. | | [LAGS\_ZEROFILL](/data-preparation#lags_zerofill) | Data Preparation | Generates lagged columns for a single variable and replaces NULL values with `0`. | | [MULTI\_LAGS](/data-preparation#multi_lags) | Data Preparation | Generates lagged columns for multiple variables at once. | | [MULTI\_LAGS\_ZEROFILL](/data-preparation#multi_lags_zerofill) | Data Preparation | Generates lagged columns for multiple variables and replaces NULL values with `0`. | | [ADD\_MONTHS](/date-and-time-functions#add_months) | Date and Time Functions | Adds the specified number of months to the date. | | [CENTURY](/date-and-time-functions#century) | Date and Time Functions | Returns the number of centuries. | | [CURDATE](/date-and-time-functions#curdate) | Date and Time Functions | Alias for CURRENT\_DATE. | | [CURRENT\_DATE](/date-and-time-functions#current_date) | Date and Time Functions | Returns the current date in the format YYYY-MM-DD. | | [CURRENT\_TIME](/date-and-time-functions#current_time) | Date and Time Functions | Returns the current time as a TIME value (e.g., `hh:mm:ss.mm`). | | [CURRENT\_TIMESTAMP](/date-and-time-functions#current_timestamp) | Date and Time Functions | Returns the current date and time as a TIMESTAMP value (e.g., YYYY-MM-DD hh🇲🇲ss.mmm). | | [DATE\_PART](/date-and-time-functions#date_part) | Date and Time Functions | Alias for EXTRACT. | | [DATE\_TRUNC](/date-and-time-functions#date_trunc) | Date and Time Functions | Returns the date or timestamp entered, truncated to the specified precision. | | [DATEADD](/date-and-time-functions#dateadd) | Date and Time Functions | Adds a specified number value (as a signed integer) to a specified date part of an input date value, and then returns that modified value. | | [DATEDIFF](/date-and-time-functions#datediff) | Date and Time Functions | This function returns an INT representing the difference between two date or time values in a specified date or time unit. | | [DAY](/date-and-time-functions#day) | Date and Time Functions | Alias for DAY\_OF\_MONTH. | | [DAY\_OF\_MONTH](/date-and-time-functions#day_of_month) | Date and Time Functions | Extracts the day-of-month portion of a timestamp or date as an integer. | | [DAY\_OF\_WEEK](/date-and-time-functions#day_of_week) | Date and Time Functions | Returns an integer, in the range of 1 to 7, that represents the day of the week. | | [DAY\_OF\_YEAR](/date-and-time-functions#day_of_year) | Date and Time Functions | Returns an integer in the range 1 to 366 that represents the day of the year. | | [DECADE](/date-and-time-functions#decade) | Date and Time Functions | The decade is the year divided by 10. | | [DOW](/date-and-time-functions#dow) | Date and Time Functions | Alias for DAY\_OF\_WEEK. | | [DOY](/date-and-time-functions#doy) | Date and Time Functions | Alias for DAY\_OF\_YEAR. | | [EOMONTH](/date-and-time-functions#eomonth) | Date and Time Functions | Returns the last day of the timestamp or date. | | [EPOCH](/date-and-time-functions#epoch) | Date and Time Functions | The number of seconds after 1970-01-01 00:00:00 UTC. | | [EXTRACT](/date-and-time-functions#extract) | Date and Time Functions | Extract a component from a timestamp or date. | | [HOUR](/date-and-time-functions#hour) | Date and Time Functions | Extracts the hour portion of a timestamp as an integer. | | [ISDATE](/date-and-time-functions#isdate) | Date and Time Functions | Returns TRUE if the input argument can be successfully cast to a date. | | [ISODOW](/date-and-time-functions#isodow) | Date and Time Functions | Extracts the day of the week based on ISO 8601, which ranges from Monday (1) to Sunday (7). | | [MAKEDATETIME](/date-and-time-functions#makedatetime) | Date and Time Functions | Returns a timestamp consisting of the specified date and time. | | [MILLISECOND](/date-and-time-functions#millisecond) | Date and Time Functions | Extracts the millisecond portion of a timestamp as an integer. | | [MINUTE](/date-and-time-functions#minute) | Date and Time Functions | Extracts the minute portion of a timestamp or date as an integer. | | [MONTH](/date-and-time-functions#month) | Date and Time Functions | Extracts the month portion of a timestamp or date as an integer. | | [MONTH\_NAME](/date-and-time-functions#month_name) | Date and Time Functions | Returns the calendar name in English of the month for the specified date. | | [MONTHS\_BETWEEN](/date-and-time-functions#months_between) | Date and Time Functions | Returns the difference between the two dates or timestamps in months as a DOUBLE. | | [MSECS](/date-and-time-functions#msecs) | Date and Time Functions | The seconds field, including fractional parts. The function multiplies the seconds part of the value by 1,000. | | [NANOS\_TO\_TIMESTAMP](/date-and-time-functions#nanos_to_timestamp) | Date and Time Functions | Convert a number of nanoseconds into a timestamp equivalent to the duration after the epoch time. | | [NEXT\_DAY](/date-and-time-functions#next_day) | Date and Time Functions | Returns the closest date after a specified date that lies on a specific day of the week. | | [NOW](/date-and-time-functions#now) | Date and Time Functions | Alias for CURRENT\_TIMESTAMP. | | [QUARTER](/date-and-time-functions#quarter) | Date and Time Functions | Returns an integer between 1 and 4 that represents the quarter of the year in which the specified date falls. | | [ROUND](/date-and-time-functions#round) | Date and Time Functions | Returns the specified date or timestamp, rounded to the specified precision. | | [SECOND](/date-and-time-functions#second) | Date and Time Functions | Extracts the seconds portion of a timestamp as an integer. | | [TIMESTAMP\_TO\_NANOS](/date-and-time-functions#timestamp_to_nanos) | Date and Time Functions | Convert timestamp into nanoseconds after epoch as BIGINT. | | [USECS](/date-and-time-functions#usecs) | Date and Time Functions | The seconds part of a time value, including fractional parts, returned as an integer. The function multiplies the seconds part of the value by 1,000,000. | | [WEEK](/date-and-time-functions#week) | Date and Time Functions | Returns the ISO-8601 week number, as an integer, of the specified timestamp or date value. | | [YEAR](/date-and-time-functions#year) | Date and Time Functions | Extracts the year portion of a timestamp or date as an integer. | | [TO\_DATE](/formatting-functions#to_date) | Formatting Functions | Converts a character value with the specified format to a DATE type. | | [TO\_NUMBER](/formatting-functions#to_number) | Formatting Functions | Converts a character value with the specified format to a DECIMAL type. | | [TO\_TIMESTAMP](/formatting-functions#to_timestamp) | Formatting Functions | Converts a character value with the specified format to a TIMESTAMP type. | | [ST\_COORDDIM](/attribute-functions#st_coorddim) | Geospatial Attribute Functions | Alias for ST\_NDIMS or ST\_NDIMENSION. Returns an INTEGER of the coordinate dimension of the specified geography. | | [ST\_DIMENSION](/attribute-functions#st_dimension) | Geospatial Attribute Functions | Returns an INTEGER that represents the dimension of the specified geography. | | [ST\_GEOMETRYTYPE](/attribute-functions#st_geometrytype) | Geospatial Attribute Functions | Returns a string representing the geometry type of the input value. | | [ST\_ISEMPTY](/attribute-functions#st_isempty) | Geospatial Attribute Functions | Returns TRUE if the specified geography value is empty, such as 'POLYGON EMPTY'. | | [ST\_MEMSIZE](/attribute-functions#st_memsize) | Geospatial Attribute Functions | Returns an INTEGER representing the number of bytes in memory required to store the specified geography. | | [ST\_NDIMENSION](/attribute-functions#st_ndims-or-st_ndimension) | Geospatial Attribute Functions | Alias for ST\_COORDDIM. | | [ST\_NDIMS](/attribute-functions) | Geospatial Attribute Functions | Alias for ST\_COORDDIM. | | [ST\_NPOINTS](/attribute-functions#st_npoints-or-st_numpoints) | Geospatial Attribute Functions | Returns an INTEGER representing the number of POINT values in a specified geography. | | [ST\_NUMPOINTS](/attribute-functions) | Geospatial Attribute Functions | Returns an INTEGER representing the number of POINT values in a specified geography. | | [ST\_SRID](/attribute-functions#st_srid) | Geospatial Attribute Functions | Returns the EPSG code of the spatial reference identifier (SRID) of the input geography. | | [ST\_X](/attribute-functions#st_x) | Geospatial Attribute Functions | Returns the x value of the specified POINT. | | [ST\_XMAX](/attribute-functions#st_xmax) | Geospatial Attribute Functions | Returns the maximum x value of the specified geography. | | [ST\_XMIN](/attribute-functions#st_xmin) | Geospatial Attribute Functions | Returns the minimum x value of the specified geography. | | [ST\_Y](/attribute-functions#st_y) | Geospatial Attribute Functions | Returns the y value of the specified POINT. | | [ST\_YMAX](/attribute-functions#st_ymax) | Geospatial Attribute Functions | Returns the maximum y value of specified geography. | | [ST\_YMIN](/attribute-functions#st_ymin) | Geospatial Attribute Functions | Returns the minimum y value of specified geography. | | [ST\_ASBINARY](/conversion-functions#st_asbinary) | Geospatial Conversion Functions | Returns the well-known binary (WKB) representation of the specified geography. Alias of ST\_ASWKB. | | [ST\_ASEWKT](/conversion-functions#st_asewkt) | Geospatial Conversion Functions | Returns a string that represents geographic coordinates of a specified POINT in the specified format. | | [ST\_ASGEOJSON](/conversion-functions#st_asgeojson) | Geospatial Conversion Functions | Alias of ST\_ASBINARY. | | [ST\_ASLATLONTEXT](/conversion-functions#st_aslatlontext) | Geospatial Conversion Functions | [Returns the GeoJSON representation of the specified geography using the IETF standards.](https://datatracker.ietf.org/doc/html/rfc7946) | | [ST\_ASTEXT](/conversion-functions#st_astext) | Geospatial Conversion Functions | Alias of ST\_ASTEXT. | | [ST\_ASWKB](/conversion-functions#st_aswkb) | Geospatial Conversion Functions | Alias of ST\_ASTEXT. | | [ST\_ASWKT](/conversion-functions#st_aswkt) | Geospatial Conversion Functions | Alias of ST\_ASWKT and ST\_EWKT. Returns the WKT representation of the specified geography. | | [ST\_GEOHASH](/conversion-functions#st_geohash) | Geospatial Conversion Functions | Returns a string that represents the geohash of the input POINT. | | [ST\_ADDPOINT](/linestring-functions#st_addpoint) | Geospatial Linestring Function | Adds a POINT to the specified LINESTRING at the specified 0-indexed location. | | [ST\_ENDPOINT](/linestring-functions#st_endpoint) | Geospatial Linestring Function | Returns the endpoint of a specified LINESTRING. The returned value is a POINT. | | [ST\_LINEFROMEWKT](/linestring-constructors#st_linefromewkt) | Geospatial Linestring Constructor | Creates a LINESTRING from the specified CHAR. | | [ST\_LINEFROMGEOJSON](/linestring-constructors#st_linefromgeojson) | Geospatial Linestring Constructor | Creates a LINESTRING represented by the specified GeoJSON. | | [ST\_LINEFROMTEXT](/linestring-constructors#st_linefromtext) | Geospatial Linestring Constructor | Creates a LINESTRING from a specified CHAR. The CHAR must be a LINESTRING value in WKT format. | | [ST\_LINEFROMWKB](/linestring-constructors#st_linefromwkb) | Geospatial Linestring Constructor | Creates a LINESTRING from the specified BINARY. The BINARY value must be a LINESTRING in WKB format. | | [ST\_LINEINTERPOLATEPOINT](/linestring-functions#st_lineinterpolatepoint) | Geospatial Linestring Function | Returns a POINT along a LINESTRING based on a specified fraction of its total length. | | [ST\_LINELOCATEPOINT](/linestring-functions#st_linelocatepoint) | Geospatial Linestring Function | Similar to ST\_LINEINTERPOLATEPOINT, this function computes a fraction based on where a specified POINT is located along the length of a specified LINESTRING. | | [ST\_LINESTRING](/linestring-constructors#st_linestring) | Geospatial Linestring Constructor | Creates a LINESTRING based on the specified inputs. | | [ST\_LINESUBSTRING](/linestring-functions#st_linesubstring) | Geospatial Linestring Function | Returns a LINESTRING that is a substring of a specified line that starts and ends at the specified fractions of its total length. | | [ST\_MAKELINE](/linestring-constructors#st_makeline) | Geospatial Linestring Constructor | Alias for ST\_LINESTRING. | | [ST\_POINTN](/linestring-functions#st_pointn) | Geospatial Linestring Function | Returns the POINT value at a specified index of the specified LINESTRING. | | [ST\_REMOVEPOINT](/linestring-functions#st_removepoint) | Geospatial Linestring Function | Removes a POINT value at a specified index from the specified line. | | [ST\_SETPOINT](/linestring-functions#st_setpoint) | Geospatial Linestring Function | Replaces a POINT value in a specified LINESTRING at a specified index. The function returns the altered LINESTRING with the replaced point. | | [ST\_STARTPOINT](/linestring-functions#st_startpoint) | Geospatial Linestring Function | Returns the starting POINT value of the line. | | [ST\_CENTROID](/point-constructors#st_centroid) | Geospatial Point Constructors | The geographic center of mass is calculated by taking the average of all points on a three-dimensional sphere, projecting the resultant point onto the sphere, and converting it back to latitude and longitude coordinates. | | [ST\_GEOGPOINT](/point-constructors#st_geogpoint) | Geospatial Point Constructors | Creates and returns a POLYGON geography with a single point, defined by the longitude and latitude specified for the function. | | [ST\_MAKEPOINT](/point-constructors#st_makepoint) | Geospatial Point Constructors | Alias for ST\_POINT. | | [ST\_POINT](/point-constructors#st_point) | Geospatial Point Constructors | Creates a POINT from the specified input arguments. | | [ST\_POINTFROMEWKT](/point-constructors#st_pointfromewkt) | Geospatial Point Constructors | Alias for ST\_POINT. Creates a POINT using an EWKT-formatted CHAR as an input argument. | | [ST\_POINTFROMGEOHASH](/point-constructors#st_pointfromgeohash) | Geospatial Point Constructors | Creates a POINT represented by the specified geohash. | | [ST\_POINTFROMGEOJSON](/point-constructors#st_pointfromgeojson) | Geospatial Point Constructors | Creates a POINT represented by the specified GeoJSON value as an input argument. | | [ST\_POINTFROMTEXT](/point-constructors#st_pointfromtext) | Geospatial Point Constructors | Alias for ST\_POINT(char). | | [ST\_POINTFROMWKB](/point-constructors#st_pointfromwkb) | Geospatial Point Constructors | Alias for ST\_POINT(binary). | | [ST\_FORCECCW](/polygon-constructors#st_forceccw) | Geospatial Polygon Constructors | Creates a standardized polygon from an existing one. defines standardized as the exterior being counterclockwise (CCW) and all holes being clockwise (CW) oriented. | | [ST\_MAKEPOLYGON](/polygon-constructors#st_makepolygon) | Geospatial Polygon Constructors | Alias for ST\_POLYGON. | | [ST\_POLYGON](/polygon-constructors#st_polygon) | Geospatial Polygon Constructors | Creates a POLYGON. | | [ST\_POLYGONFROMEWKT](/polygon-constructors#st_polygonfromewkt) | Geospatial Polygon Constructors | Creates a POLYGON using an EWKT-formatted CHAR as an input argument. Alias for the ST\_POLYGON constructor. | | [ST\_POLYGONFROMGEOJSON](/polygon-constructors#st_polygonfromgeojson) | Geospatial Polygon Constructors | Creates a POLYGON from the specified POINT, POINT array, LINESTRING, or POLYGON geography. | | [ST\_POLYGONFROMTEXT](/polygon-constructors#st_polygonfromtext) | Geospatial Polygon Constructors | Alias for ST\_POLYGON(char). | | [ST\_POLYGONFROMWKB](/polygon-constructors#st_polygonfromwkb) | Geospatial Polygon Constructors | Alias for ST\_POLYGON(binary). | | [ST\_WHOLEEARTH](/polygon-constructors#st_wholeearth) | Geospatial Polygon Constructors | Returns the database internal representation of the whole earth polygon. | | [ST\_ANGLE](/spatial-measurement#st_angle) | Geospatial Spatial Measurement | Calculates the angle between two lines. | | [ST\_AREA](/spatial-measurement#st_area) | Geospatial Spatial Measurement | Returns the area of the specified geospatial object in the specified unit of measurement. | | [ST\_AZIMUTH](/spatial-measurement#st_azimuth) | Geospatial Spatial Measurement | Returns the azimuth of the line from `point1` to `point2` in radians. | | [ST\_DISTANCE](/spatial-measurement#st_distance) | Geospatial Spatial Measurement | Returns the minimum distance between the specified arguments. | | [ST\_DISTANCESPHERE](/spatial-measurement#st_distancesphere) | Geospatial Spatial Measurement | Returns the minimum distance between the specified arguments using a spherical computation. | | [ST\_DISTANCESPHEROID](/spatial-measurement#st_distancespheroid) | Geospatial Spatial Measurement | Returns the minimum distance between the specified arguments using a spheroid computation. | | [ST\_EUCLIDEANDISTANCE3D](/spatial-measurement#st_euclideandistance3d) | Geospatial Spatial Measurement | Returns the minimum distance between two geospatial points with altitude values. | | [ST\_HAUSDORFFDISTANCE](/spatial-measurement#st_hausdorffdistance) | Geospatial Spatial Measurement | Returns the Hausdorff distance between two geographies in a specified measurement. | | [ST\_LENGTH](/spatial-measurement#st_length) | Geospatial Spatial Measurement | Returns the length of the specified line in the specified measurement unit. | | [ST\_LENGTH2D](/spatial-measurement#st_length2d) | Geospatial Spatial Measurement | Alias for ST\_LENGTH. | | [ST\_MAXDISTANCE](/spatial-measurement#st_maxdistance) | Geospatial Spatial Measurement | Returns maximum distance between the specified arguments. | | [ST\_MINIMUMDISTANCETOSURFACE](/spatial-measurement#st_minimumdistancetosurface) | Geospatial Spatial Measurement | Calculates the shortest distance between any point along a Euclidean line segment in three-dimensional space and the surface of the Earth. | | [ST\_PERIMETER](/spatial-measurement#st_perimeter) | Geospatial Spatial Measurement | Returns the length of the exterior (outer ring) of the POLYGON in the specified unit of measurement. | | [ST\_PERIMETER2D](/spatial-measurement#st_perimeter2d) | Geospatial Spatial Measurement | Alias for ST\_PERIMETER. | | [ST\_BOUNDINGDIAGONAL](/spatial-operators#st_boundingdiagonal) | Geospatial Spatial Operators | Returns the diagonal LINESTRING from the minimum point to the maximum point of the bounding box that ST\_ENVELOPE returns. | | [ST\_BUFFER](/spatial-operators#st_buffer) | Geospatial Spatial Operators | Returns a geography that contains all points where the distance from the geography is less than or equal to the specified distance. | | [ST\_CLOSESTPOINT](/spatial-operators#st_closestpoint) | Geospatial Spatial Operators | Returns the two-dimensional POINT of one specified geospatial object that is closest to a second specified geospatial object. | | [ST\_CONVEXHULL](/spatial-operators#st_convexhull) | Geospatial Spatial Operators | The convex hull is the smallest convex geometry that encloses the input geometry. | | [ST\_DIFFERENCEARRAY](/spatial-operators#st_differencearray) | Geospatial Spatial Operators | Returns an array containing any geospatial objects that are present in the first specified geospatial argument that are not found in the second geospatial argument. | | [ST\_ENVELOPE](/spatial-operators#st_envelope) | Geospatial Spatial Operators | Returns a POLYGON that represents the minimum bounding box for the specified geography. | | [ST\_EXPAND](/spatial-operators#st_expand) | Geospatial Spatial Operators | Returns the bounding box of a specified geospatial value, which is expanded by a specified length. | | [ST\_EXTERIORRING](/spatial-operators#st_exteriorring) | Geospatial Spatial Operators | Returns a LINESTRING that represents the exterior ring of a provided POLYGON value. | | [ST\_FLIPCOORDINATES](/spatial-operators#st_flipcoordinates) | Geospatial Spatial Operators | Returns a new geographic object with the X and Y coordinates switched using the specified argument. | | [ST\_FORCE2D](/spatial-operators#st_force2d) | Geospatial Spatial Operators | Convert a geographic object into a two-dimensional geography. | | [ST\_INTERIORRINGN](/spatial-operators#st_interiorringn) | Geospatial Spatial Operators | Returns a LINESTRING representing the interior ring of the specified POLYGON, which is specified by its index. (1-indexed) | | [ST\_INTERSECTALL](/spatial-operators#st_intersectall) | Geospatial Spatial Operators | Returns the intersection of all geographies in the specified array. All geographies in the array must be the same type. | | [ST\_INTERSECTIONARRAY](/spatial-operators#st_intersectionarray) | Geospatial Spatial Operators | Returns a geography that represents the point-set intersection of two geographies. | | [ST\_LONGESTLINE](/spatial-operators#st_longestline) | Geospatial Spatial Operators | Returns the longest LINESTRING between two specified geospatial arguments. | | [ST\_MAKEENVELOPE](/spatial-operators#st_makeenvelope) | Geospatial Spatial Operators | Returns a POLYGON with vertices that represent the minimum bounding box for the specified coordinates. | | [ST\_MINIMUMBOUNDINGCIRCLE](/spatial-operators#st_minimumboundingcircle) | Geospatial Spatial Operators | Returns the smallest circle POLYGON that contains the specified geographic object. | | [ST\_MULTIDIFFERENCEARRAY](/spatial-operators#st_multidifferencearray) | Geospatial Spatial Operators | Returns an array of geographies that represents the parts of the union of the geographies in the first array that do not intersect with the union of geographies in the second array. | | [ST\_MULTIINTERSECTIONARRAY](/spatial-operators#st_multiintersectionarray) | Geospatial Spatial Operators | When you specify two arrays of geospatial objects, this function returns an array of any intersections. | | [ST\_MULTISYMDIFFERENCEARRAY](/spatial-operators#st_multisymdifferencearray) | Geospatial Spatial Operators | When you specify two arrays of geospatial objects, this function returns an array of any geospatial values that do not intersect. | | [ST\_MULTIUNIONARRAY](/spatial-operators#st_multiunionarray) | Geospatial Spatial Operators | When you specify two arrays of geospatial objects, this function returns one array that represents a union of all geographies in both arrays. | | [ST\_NRINGS](/spatial-operators#st_nrings) | Geospatial Spatial Operators | Returns the number of rings of the specified POLYGON, including both interior and exterior rings. | | [ST\_NUMINTERIORRING](/spatial-operators#st_numinteriorrings-or-st_numinteriorring) | Geospatial Spatial Operators | Returns the number of interior rings of the specified POLYGON. | | [ST\_NUMINTERIORRINGS](/spatial-operators) | Geospatial Spatial Operators | Returns the number of interior rings of the specified POLYGON. | | [ST\_POINTONSURFACE](/spatial-operators#st_pointonsurface) | Geospatial Spatial Operators | Returns a POINT guaranteed to intersect the specified geospatial object. | | [ST\_PROJECT](/spatial-operators#st_project) | Geospatial Spatial Operators | Returns a POINT by projecting a distance and an azimuth value from the specified starting POINT value. | | [ST\_REDUCEPRECISION](/spatial-operators#st_reduceprecision) | Geospatial Spatial Operators | Returns a new geospatial object with all POINT values rounded to the specified decimal precision. | | [ST\_REMOVEREPEATEDPOINTS](/spatial-operators#st_removerepeatedpoints) | Geospatial Spatial Operators | Returns a new geospatial object with no repeated POINT values. | | [ST\_REVERSE](/spatial-operators#st_reverse) | Geospatial Spatial Operators | Returns a new geospatial object with the vertexes reversed. | | [ST\_SEGMENTIZE](/spatial-operators#st_segmentize) | Geospatial Spatial Operators | Returns a geospatial object that the function modifies to have no segment longer than the specified max\_segment\_length in meters. | | [ST\_SHORTESTLINE](/spatial-operators#st_shortestline) | Geospatial Spatial Operators | Returns the shortest LINESTRING between two specified geospatial arguments. | | [ST\_SIMPLIFY](/spatial-operators#st_simplify) | Geospatial Spatial Operators | Returns a simplified version of the specified geography, which is either a POINT or LINESTRING. | | [ST\_SIMPLIFYARRAY](/spatial-operators#st_simplifyarray) | Geospatial Spatial Operators | Returns a POLYGON array that represents a simplified version of the specified geography, which is either a POINT, LINESTRING, or POLYGON. | | [ST\_SNAPTOGRID](/spatial-operators#st_snaptogrid) | Geospatial Spatial Operators | Returns a new geography value with all POINT values rounded to the specified precisions. | | [ST\_SYMDIFFERENCEARRAY](/spatial-operators#st_symdifferencearray) | Geospatial Spatial Operators | Returns a geographic array that contains the parts that are not common between two geographic objects, geo1 and geo2. | | [ST\_UNIONARRAY](/spatial-operators#st_unionarray) | Geospatial Spatial Operators | Performs a union of the input geography values to produce a geographic array. | | [ST\_CLUSTERDBSCAN](/spatial-relationships#st_clusterdbscan) | Geospatial Spatial Relationships | [Returns the cluster number for each input geography, based on a two-dimensional implementation of the density-based spatial clustering of applications with noise (DBSCAN) algorithm.](https://en.wikipedia.org/wiki/DBSCAN) | | [ST\_CONTAINS](/spatial-relationships#st_contains) | Geospatial Spatial Relationships | Returns TRUE if the first geographic argument, geo1, contains the second geographic argument, geo2. | | [ST\_CONTAINSPROPERLY](/spatial-relationships#st_containsproperly) | Geospatial Spatial Relationships | Returns true if geo2 lies entirely in the interior of geo1 and does not intersect or touch the boundary or exterior points. | | [ST\_COVEREDBY](/spatial-relationships#st_coveredby) | Geospatial Spatial Relationships | Returns TRUE if no POINT in geo1 is outside of geo2. | | [ST\_COVERS](/spatial-relationships#st_covers) | Geospatial Spatial Relationships | Returns TRUE if no POINT in geo2 is outside of geo1. | | [ST\_CROSSES](/spatial-relationships#st_crosses) | Geospatial Spatial Relationships | Returns TRUE if two geospatial objects meet these criteria: The intersection of the geospatial interiors is not empty. The intersection is not equal to geo1 or geo2. Neither geospatial object is a single POINT. | | [ST\_DISJOINT](/spatial-relationships#st_disjoint) | Geospatial Spatial Relationships | Returns true if the specified geographies have no intersection, including boundaries. Both geographic arguments can be different types. | | [ST\_DWITHIN](/spatial-relationships#st_dwithin) | Geospatial Spatial Relationships | Returns TRUE if the geographies are within a specified distance in meters. | | [ST\_EQUALS](/spatial-relationships#st_equals) | Geospatial Spatial Relationships | Returns TRUE if both geographies are spatially equal. | | [ST\_INTERSECTS](/spatial-relationships#st_intersects) | Geospatial Spatial Relationships | Returns TRUE if the specified geographies have intersection, including boundaries. | | [ST\_ISCCW](/spatial-relationships#st_isccw) | Geospatial Spatial Relationships | Alias for ST\_ISPOLYGONCCW. | | [ST\_ISCLOSED](/spatial-relationships#st_isclosed) | Geospatial Spatial Relationships | Returns TRUE if an input POLYGON has an exterior that is counter-clockwise. | | [ST\_ISPOLYGONCCW](/spatial-relationships#st_ispolygonccw) | Geospatial Spatial Relationships | Returns TRUE if an input LINESTRING has starting and ending points that are equal. | | [ST\_ISPOLYGONCW](/spatial-relationships#st_ispolygoncw) | Geospatial Spatial Relationships | Returns TRUE if an input POLYGON has an exterior that is clockwise. | | [ST\_ISRING](/spatial-relationships#st_isring) | Geospatial Spatial Relationships | Returns TRUE if the specified LINESTRING is closed and does not intersect itself. | | [ST\_ISSIMPLE](/spatial-relationships#st_issimple) | Geospatial Spatial Relationships | For LINESTRING values, this function returns FALSE if any line segments intersect anywhere besides the endpoints. For POLYGON values, the function returns FALSE if either the exterior ring or any interior hole is not simple. | | [ST\_ISVALID](/spatial-relationships#st_isvalid) | Geospatial Spatial Relationships | [Returns TRUE if the specified geospatial value is a well-formed and valid geography according to the OGC standards.](https://www.ogc.org/standard/sfa/) | | [ST\_OVERLAPS](/spatial-relationships#st_overlaps) | Geospatial Spatial Relationships | Returns TRUE if both geographic arguments are of the same dimension and they intersect each other, but neither contains the other. | | [ST\_POINTINSIDECIRCLE](/spatial-relationships#st_pointinsidecircle) | Geospatial Spatial Relationships | Returns TRUE if the geographic object is inside a circle that is centered at the specified point coordinates and with the specified radius. | | [ST\_RELATE](/spatial-relationships#st_relate) | Geospatial Spatial Relationships | Returns the [DE-9IM](https://postgis.net/workshops/postgis-intro/de9im.html) intersection string that represents the nature of the intersection with the specified geographies. | | [ST\_TOUCHES](/spatial-relationships#st_touches) | Geospatial Spatial Relationships | Returns TRUE if the only POINT values in common between the two geographic arguments lie in the union of their boundaries. | | [ST\_WITHIN](/spatial-relationships#st_within) | Geospatial Spatial Relationships | Alias for ST\_CONTAINS. | | [ST\_DISTANCE](/spatiotemporal-measurement#st_distance) | Geospatial Spatiotemporal Measurement | Returns the two-dimensional interpolated minimum simultaneous distance between two LINESTRING-TIMESTAMP array pairs in the specified unit of measurement. | | [ST\_MAXDISTANCE](/spatiotemporal-measurement#st_maxdistance) | Geospatial Spatiotemporal Measurement | Returns the two-dimensional interpolated maximum cotemporal distance between two LINESTRING-TIMESTAMP array pairs in the specified unit of measurement. | | [ST\_TOTALSECONDSININTERSECTION](/spatiotemporal-measurement#st_totalsecondsinintersection) | Geospatial Spatiotemporal Measurement | Returns the total number of seconds spent in the intersection result calculated by the spatiotemporal version of ST\_INTERSECTION. | | [ST\_INTERSECTION](/spatiotemporal-operators#st_intersection) | Geospatial Spatiotemporal Operators | Returns a tuple that represents the intersection of a spatiotemporal LINESTRING with a static geography. | | [ST\_LINEGETALLTIMESATPOINT](/spatiotemporal-operators#st_linegetalltimesatpoint) | Geospatial Spatiotemporal Operators | Returns a timestamp array of all times when the specified LINESTRING value intersects the specified POINT value. | | [ST\_LINEGETPOINTATTIME](/spatiotemporal-operators#st_linegetpointattime) | Geospatial Spatiotemporal Operators | Returns a POINT within the bounds of the specified LINESTRING that corresponds to the interpolated point at the specified TIMESTAMP value. | | [ST\_LINEGETTIMEATPOINT](/spatiotemporal-operators#st_linegettimeatpoint) | Geospatial Spatiotemporal Operators | Returns the interpolated time of the specified POINT on the specified LINESTRING that is paired with a TIMESTAMP ARRAY. | | [ST\_LONGESTLINE](/spatiotemporal-operators#st_longestline) | Geospatial Spatiotemporal Operators | With the specified two LINESTRING-TIMESTAMP ARRAY pairs, this function returns a two-point LINESTRING that represents the maximum distance between points at a concurrent time. | | [ST\_SHORTESTLINE](/spatiotemporal-operators#st_shortestline) | Geospatial Spatiotemporal Operators | When you specify two LINESTRING-TIMESTAMP ARRAY pairs, this function returns a LINESTRING with two points that represents the minimum distance between points at a concurrent time. | | [HLL\_SKETCH\_CREATE](/hyperloglog-functions#hll_sketch_create) | HyperLogLog Functions | Creates an HLL sketch from the data on a specified aggregated column. Returns a HASH((2^log2k) + 8) data representation of the sketch that you can store in a separate column. | | [HLL\_SKETCH\_GET\_ESTIMATE](/hyperloglog-functions#hll_sketch_get_estimate) | HyperLogLog Functions | The scalar function converts a sketch into a distinct count estimate of a sketch value. Returns the distinct count estimate as a BIGINT. | | [HLL\_SKETCH\_GET\_ESTIMATE\_BOUND](/hyperloglog-functions#hll_sketch_get_estimate_bound) | HyperLogLog Functions | Takes a HLL\_SKETCH column or an integral log2k literal value and returns the resulting bounding 95-percent confidence interval error proportion as a DOUBLE. | | [HLL\_SKETCH\_TO\_STRING](/hyperloglog-functions#hll_sketch_to_string) | HyperLogLog Functions | The HLL\_SKETCH\_TO\_STRING scalar function takes a HLL\_SKETCH column or value and returns a string summary of the sketch. | | [HLL\_SKETCH\_UNION (aggregate function)](/hyperloglog-functions#hll_sketch_union-aggregate-function) | HyperLogLog Functions | Merges multiple sketches in a single column into a unified sketch. All sketches must have the same precision. This function is an aggregate function and operates on a column. | | [HLL\_SKETCH\_UNION (scalar function)](/hyperloglog-functions#hll_sketch_union-scalar-function) | HyperLogLog Functions | Merges two sketches into a new combined sketch. This function is a scalar function and operates row-wise. The scalar function merges two sketch columns with heterogeneous precisions into a sketch with the lower of the two precisions. | | [ABS](/math-functions-and-operators#abs) | Math Functions | Returns the absolute value of a specified floating-point number. | | [ACOS](/math-functions-and-operators#acos) | Math Functions | Returns the inverse cosine of a specified floating-point number. | | [ACOSH](/math-functions-and-operators#acosh) | Math Functions | Returns the hyperbolic arc-cosine of a specified floating-point number. | | [ASIN](/math-functions-and-operators#asin) | Math Functions | Returns the inverse sine of a specified floating-point number. | | [ASINH](/math-functions-and-operators#asinh) | Math Functions | Returns the hyperbolic arc-sine of a specified floating-point number. | | [ATAN](/math-functions-and-operators#atan) | Math Functions | Returns the inverse tangent of a specified floating-point number. | | [ATAN2](/math-functions-and-operators#atan2) | Math Functions | Returns the inverse tangent of two numeric, floating-point values. | | [ATANH](/math-functions-and-operators#atanh) | Math Functions | Returns the hyperbolic arc-tangent of a specified floating-point number. | | [BICDF](/math-functions-and-operators#bicdf) | Math Functions | The cumulative distribution function of the standard bivariate normal distribution. | | [BIPDF](/math-functions-and-operators#bipdf) | Math Functions | The probability density function of the standard bivariate normal distribution. | | [BITAND](/math-functions-and-operators#bitand) | Math Functions | Alias for the BITFUNC syntax `BITFUNC('AND', x, y)`. | | [BITFUNC](/math-functions-and-operators#bitfunc) | Math Functions | Performs a variety of bit operations. Can be any of these string literals: `'AND'`, `'OR'`, or `'XOR'`. | | [BITNOT](/math-functions-and-operators#bitnot) | Math Functions | Returns the bitwise negation of integral\_x. | | [BITOR](/math-functions-and-operators#bitor) | Math Functions | Alias for the BITFUNC syntax BITFUNC('OR', x, y). | | [BITXOR](/math-functions-and-operators#bitxor) | Math Functions | Alias for the BITFUNC syntax BITFUNC('XOR', x, y). | | [BOOLAND](/math-functions-and-operators#booland) | Math Functions | Alias for the BOOLFUNC syntax BOOLFUNC('AND', x, y). | | [BOOLFUNC](/math-functions-and-operators#boolfunc) | Math Functions | Performs a Boolean logical evaluation on arguments x and y. Can be any of these string literals: `'AND'`, `'OR'`, or `'XOR'`. | | [BOOLNOT](/math-functions-and-operators#boolnot) | Math Functions | Returns the logical negation of the BOOLFUNC function. | | [BOOLOR](/math-functions-and-operators#boolor) | Math Functions | Alias for the BOOLFUNC syntax BOOLFUNC('OR', x, y). | | [BOOLXOR](/math-functions-and-operators#boolxor) | Math Functions | Alias for the BOOLFUNC syntax BOOLFUNC('XOR', x, y). | | [CBRT](/math-functions-and-operators#cbrt) | Math Functions | Returns the cube root of the numeric value x. | | [CDF](/math-functions-and-operators#cdf) | Math Functions | The cumulative distribution function of the standard normal distribution. Returns the probability that a random sample is less than or equal to the specified value. | | [CEIL](/math-functions-and-operators#ceil) | Math Functions | Returns the nearest integer greater than or equal to x. | | [CEILING](/math-functions-and-operators#ceiling) | Math Functions | Alias for CEIL. | | [COS](/math-functions-and-operators#cos) | Math Functions | Returns the cosine of x. | | [COSH](/math-functions-and-operators#cosh) | Math Functions | Returns the hyperbolic cosine of x. | | [COT](/math-functions-and-operators#cot) | Math Functions | Returns the cotangent of x. | | [DEGREES](/math-functions-and-operators#degrees) | Math Functions | Returns the corresponding angle in degrees for x in radians. | | [DIV](/math-functions-and-operators#div) | Math Functions | Returns the result of x divided by y. If y is zero, returns NULL. | | [ERF](/math-functions-and-operators#erf) | Math Functions | The error function is used for measurements that follow a normal distribution. | | [ERFC](/math-functions-and-operators#erfc) | Math Functions | The complement of the error function. ERFC(x) = 1 - ERF(x). | | [EXP](/math-functions-and-operators#exp) | Math Functions | Returns the exponential of x (e raised to the power of x). | | [FLOOR](/math-functions-and-operators#floor) | Math Functions | Returns the nearest integer less than or equal to x. | | [GAMMA](/math-functions-and-operators#gamma) | Math Functions | [Gamma function.](https://en.wikipedia.org/wiki/Gamma_function) | | [HEXBINX](/math-functions-and-operators#hexbinx) | Math Functions | Returns the x-coordinate of the center of the nearest hexagonal bin to the point (x, y). | | [HEXBINY](/math-functions-and-operators#hexbiny) | Math Functions | Returns the y-coordinate of the center of the nearest hexagonal bin to the point (x, y). | | [IERF](/math-functions-and-operators#ierf) | Math Functions | The inverse of the ERF error function. | | [IERFC](/math-functions-and-operators#ierfc) | Math Functions | The inverse of the complement of the error function. | | [LEAKYRELU](/math-functions-and-operators#leakyrelu) | Math Functions | Returns the leaky rectified linear unit function of x. | | [LEFT\_SHIFT](/math-functions-and-operators#left_shift) | Math Functions | Returns x shifted to the left by y bits. | | [LN](/math-functions-and-operators#ln) | Math Functions | Returns the natural logarithm of x. | | [LOG](/math-functions-and-operators#log) | Math Functions | Returns the base 10 logarithm of x. The optional base argument specifies the numeral system to use. If unspecified, the function defaults to base 10. | | [LOG\_GAMMA](/math-functions-and-operators#log_gamma) | Math Functions | The natural logarithm of the absolute value of the gamma function. | | [LOG2](/math-functions-and-operators#log2) | Math Functions | Returns the base 2 logarithm of x. | | [MOD](/math-functions-and-operators#mod) | Math Functions | Returns the remainder from x divided by y. | | [PI](/math-functions-and-operators#pi) | Math Functions | Returns the constant value of π. | | [PMOD](/math-functions-and-operators#pmod) | Math Functions | Returns the smallest non-negative equivalence class of x % y. | | [POWER](/math-functions-and-operators#power) | Math Functions | Returns x raised to the power of y. | | [PROBIT](/math-functions-and-operators#probit) | Math Functions | The inverse of the cumulative distribution function. | | [RADIANS](/math-functions-and-operators#radians) | Math Functions | Returns the corresponding angle in radians for x in degrees. | | [RAND](/math-functions-and-operators#rand) | Math Functions | Takes no argument and returns a random DOUBLE value in the range \[0, 1). | | [RELU](/math-functions-and-operators#relu) | Math Functions | Returns the rectified linear unit function of x. | | [RIGHT\_SHIFT](/math-functions-and-operators#right_shift) | Math Functions | Returns x shifted to the right by y bits. | | [ROUND](/math-functions-and-operators#round) | Math Functions | Returns x rounded to the nearest integer. | | [SIGN](/math-functions-and-operators#sign) | Math Functions | Returns the positive (+1), zero (0), or negative (-1) sign of x. | | [SIN](/math-functions-and-operators#sin) | Math Functions | Returns the sine of x. | | [SINH](/math-functions-and-operators#sinh) | Math Functions | Returns the hyperbolic sine of x. | | [SQRT](/math-functions-and-operators#sqrt) | Math Functions | Returns the square root of x. | | [SQUARE](/math-functions-and-operators#square) | Math Functions | Returns the square of x. | | [TAN](/math-functions-and-operators#tan) | Math Functions | Returns the tangent of x. | | [TANH](/math-functions-and-operators#tanh) | Math Functions | Returns the hyperbolic tangent of x. | | [TO\_BASE](/math-functions-and-operators#to_base) | Math Functions | Converts an integer value to its string representation in a specified base (radix). | | [TRUNC](/math-functions-and-operators#trunc) | Math Functions | Returns x truncated to y decimal places. | | [TRUNCATE](/math-functions-and-operators#truncate) | Math Functions | Alias for TRUNC. | | [ABS](/matrix-functions-and-operators#matrix-functions) | Matrix Functions | Returns magnitude of one-dimensional matrix or vector. | | [CROSS\_ENTROPY\_LOSS](/matrix-functions-and-operators) | Matrix Functions | Returns the cross entropy loss of two one-dimensional matrices or vectors. | | [DET](/matrix-functions-and-operators) | Matrix Functions | Returns the determinant of the matrix as a double. | | [DOT](/matrix-functions-and-operators) | Matrix Functions | Returns dot product of two one-dimensional matrices/vectors. | | [EIGEN](/matrix-functions-and-operators) | Matrix Functions | Returns eigenvalues and eigenvalues of a square matrix as a vector of pairs. | | [FROBENIUS](/matrix-functions-and-operators) | Matrix Functions | Returns the Frobenius norm of a matrix. | | [HINGE\_LOSS](/matrix-functions-and-operators) | Matrix Functions | Returns the hinge loss of two one-dimensional matrices or vectors. | | [IDENTITY\_MATRIX](/matrix-functions-and-operators) | Matrix Functions | Returns an identity matrix of the specified dimension. | | [INVERSE](/matrix-functions-and-operators) | Matrix Functions | Returns inverse of a square, invertible matrix. | | [LOG\_LOSS](/matrix-functions-and-operators) | Matrix Functions | Returns the log loss of two one-dimensional matrices or vectors. | | [LOGITS\_LOSS](/matrix-functions-and-operators) | Matrix Functions | Returns the logits loss of two one-dimensional matrices or vectors. | | [LUPQ\_DECOMP](/matrix-functions-and-operators) | Matrix Functions | Returns LUPQ decomposition of a square matrix A as a tuple of 4 matrices where PAQ = LU. | | [MATRIX\_DIM](/matrix-functions-and-operators) | Matrix Functions | Returns the dimensions of the specified matrix as a tuple of (row, col) integers. | | [MAKE\_MATRIX\_IXJ](/matrix-functions-and-operators) | Matrix Functions | Creates an `ixj` matrix with elements `e_00`, `…`, `e_ij`. | | [MATRIX\_FROM\_TEXT](/matrix-functions-and-operators) | Matrix Functions | Creates a matrix from the specified string. | | [MATRIX\_TRACE](/matrix-functions-and-operators) | Matrix Functions | Returns trace of a square matrix as a double. | | [NULL\_MATRIX](/matrix-functions-and-operators) | Matrix Functions | Returns a NULL matrix of the specified (row, col) dimensions. | | [QR\_DECOMP](/matrix-functions-and-operators) | Matrix Functions | Returns QR decomposition of a matrix as a tuple of 2 matrices. | | [SOFTMAX](/matrix-functions-and-operators) | Matrix Functions | Returns the softmax of a one-dimensional matrix or vector. | | [SVD\_DECOMP](/matrix-functions-and-operators) | Matrix Functions | Returns SVD decomposition of a matrix as a tuple of 3 matrices. | | [TRANSPOSE](/matrix-functions-and-operators) | Matrix Functions | Returns transpose of the matrix. | | [VECTOR\_ARGMAX](/matrix-functions-and-operators) | Matrix Functions | Returns the argmax of a one-dimensional matrix or vector. | | [VECTOR\_ARGMIN](/matrix-functions-and-operators) | Matrix Functions | Returns the argmin of a one-dimensional matrix or vector. | | [VECTOR\_MAX](/matrix-functions-and-operators) | Matrix Functions | Returns the maximum of elements in a one-dimensional matrix/vector. | | [VECTOR\_MIN](/matrix-functions-and-operators) | Matrix Functions | Returns the minimum of elements in a one-dimensional matrix/vector. | | [VECTOR\_SUM](/matrix-functions-and-operators) | Matrix Functions | Returns the sum of elements in a one-dimensional matrix/vector. | | [ZERO\_MATRIX](/matrix-functions-and-operators) | Matrix Functions | Returns a zero matrix of the given (row, col) dimensions. | | [IP](/network-type-functions#ip) | Network Type Functions | Casts an IP data type from an IPV4 data type expression. | | [IPV4](/network-type-functions#ipv4) | Network Type Functions | Casts an IPV4 address from an IPV6 address. | | [IS\_IPV4](/network-type-functions#is_ipv4) | Network Type Functions | Tests whether the database can convert the `IP` value to the `IPV4` data type. | | [SUBNET](/network-type-functions#subnet) | Network Type Functions | Computes the prefix from an `IP` or `IPV4` value and the size of the prefix. | | [CANCEL](/query-management#cancel) | Query Management | Cancels a running query based on its specific query identifier. | | [KILL](/query-management#kill) | Query Management | Kills a running query identified by its UUID. | | [BIGINT](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type `bigint`. | | [BINARY](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses a hexadecimal string (such as 0x54ab) to create a binary value. Letters can be of either case. | | [BOOLEAN](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses a string to create a Boolean value. The string must contain either true or false. It is case-insensitive. | | [BYTE](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type byte. | | [CHAR](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Creates a string version of the numeric value. | | [DATE](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses a string in the form 'YYYY-MM-DD' to create a date. Extra characters are ignored. | | [DAYS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type days to be used in date calculations. | | [DECIMAL](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type decimal. | | [DOUBLE](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type double. | | [FLOAT](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type float. | | [HASH](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Creates a fixed length binary value, with length ``, from a string, i.e. `0x1234abcd`. Zero extended if the string does not have enough bytes, truncated if it has too many. | | [HOURS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type hours. | | [INTEGER](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type integer. | | [MICROSECONDS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type microseconds. | | [MILLISECONDS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type milliseconds. | | [MINUTES](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type minutes. | | [MONTHS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type months to be used in date calculations. | | [NANOSECONDS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type nanoseconds. | | [SECONDS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type seconds. | | [SMALLINT](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Casts the argument to a value of type `smallint`. | | [TIME](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses the string to create a time value. The string must be in the form `'HH:MM[.SSSSSSSSS]'`. | | [TIMESTAMP](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses a string and makes a timestamp value. The string must be in the format `'YYYY-MM-DD[ HH:MM][.SSSSSSSSS]'`. | | [UUID](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Parses the string and makes a UUID value. The string must be a valid UUID. | | [WEEKS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type weeks to be used in date calculations. | | [YEARS](/scalar-data-conversion-functions) | Scalar Data Conversion Functions | Converts an integral value to an interval value of type years. | | [ARRAY\_CAP](/transform-data-in-data-pipelines#array-data-transformation-functions) | Special Data Pipeline Transformation Functions | Restrict the length of an array to a maximum number of elements. | | [ARRAY\_COMPACT](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Removes NULL values from the array. | | [ARRAY\_CONTAINS](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Returns `true` if the array contains the specified value. | | [ARRAY\_SORT](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Sort and return the input array based on the natural ordering of its elements or the specified Lambda function. | | [CASE WHEN](/transform-data-in-data-pipelines#special-data-pipeline-transformation-functions) | Special Data Pipeline Transformation Functions | Returns the `result` value based on whether an expression is `true`. | | [ELEMENT\_AT](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Returns the element value of the array or tuple at the specified index. | | [EXPLODE\_OUTER](/special-data-pipeline-transformation-functions#explode_outer) | Special Data Pipeline Transformation Functions | Expands a one-dimensional or multidimensional array into its elements with one element per row of output from the system. | | [FILTER](/special-data-pipeline-transformation-functions#filter) | Special Data Pipeline Transformation Functions | Filters elements in an array based on the logic in a lambda expression. | | [FLATTEN](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Transforms an `N`-dimensional array into an `N-1`-dimensional array. | | [IF](/transform-data-in-data-pipelines#logical-operations-transformation-functions) | Special Data Pipeline Transformation Functions | Returns `T` if the expression `X` evaluates to `true`, or the function returns `F` if `X` evaluates to `false`. | | [LOOKUP](/load-data-from-external-sources-in-data-pipelines) | Special Data Pipeline Transformation Functions | Look up and load data in an external data source. | | [MAP\_KEYS](/transform-data-in-data-pipelines#other-data-transformation-functions) | Special Data Pipeline Transformation Functions | Returns the keys in the specified JSON string. | | [MAP\_VALUES](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Returns the values in the specified JSON string. | | [METADATA](/load-metadata-and-file-based-partitioned-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Extracts the metadata value for the specified key from available metadata for the pipeline. | | [PARSE\_DELIMITED\_ARRAY](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Converts a string of text data representing an array into a `CHAR[]`. | | [REDUCE](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Applies a merge function to a starting value and all elements in the array, and then reduces the array to a single value. Optionally, specify a finish function for the returned single value. | | [TO\_ARRAY\_LENGTH](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Create the specified number of copies of any JSON object in an array. | | [TRANSFORM](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Transforms an array based on the logic in a lambda expression. | | [WIDTH\_BUCKET](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Creates `N` equal-width buckets in the range `[min,max)` as a histogram. | | [ZIP\_WITH](/transform-data-in-data-pipelines) | Special Data Pipeline Transformation Functions | Merges two or more input arrays, element-wise, into a single array using the trailing combining function. | | [RECORD\_UUID](/lat-transformation-functions#record_uuid) | Other Transformation Functions | Returns string that represents a unique identifier for the record in a specified pipeline for a specified file\_group or topic. | | [COMMIT](/other-functions-and-expressions#commit) | Other Functions and Expressions | Returns the most recent commit hash of the database to which the client is currently connected. | | [RAND\_UUID](/other-functions-and-expressions#rand_uuid) | Other Functions and Expressions | Generates a random UUID value (version 4). | | [TYPE\_STRIP](/other-functions-and-expressions#type_strip) | Other Functions and Expressions | Returns the SQL type of the specified value. | | [UUID\_GENERATE](/other-functions-and-expressions#uuid_generate) | Other Functions and Expressions | Generates a random UUID value (version 4). | | [CURRENT\_DATABASE](/other-functions-and-expressions#current_database) | System Functions | Alias for DATABASE. | | [CURRENT\_GROUPS](/other-functions-and-expressions#current_groups) | System Functions | Returns the fully-qualified names of groups in the database. | | [CURRENT\_NODE](/other-functions-and-expressions#current_node) | System Functions | Returns the name of the SQL Node where the current query executes. The name of the node corresponds to the name column in the `sys.nodes` system catalog table. | | [CURRENT\_NODE\_ID](/other-functions-and-expressions#current_node_id) | System Functions | Returns the identifier of the SQL Node where the current query executes. | | [CURRENT\_SCHEMA](/other-functions-and-expressions#current_schema) | System Functions | Returns the name of the current schema. | | [CURRENT\_SESSION\_ID](/other-functions-and-expressions#current_session_id) | System Functions | Returns the Universally Unique IDentifier (UUID) of the current session. | | [CURRENT\_SYSTEM](/other-functions-and-expressions#current_system) | System Functions | Returns the name of the system. | | [CURRENT\_USER](/other-functions-and-expressions#current_user) | System Functions | Returns the user for the current connection. | | [DATABASE](/other-functions-and-expressions#database) | System Functions | Returns the name of the database to which the client is currently connected. | | [SHOW](/other-functions-and-expressions#show) | System Functions | The SHOW function enables you to explore the database and its metadata for user-defined items. | | [VERSION](/other-functions-and-expressions#version) | System Functions | Returns the version of the database to which the client is currently connected. | | [CONVERT\_LOCAL\_TIMESTAMP\_TO\_UTC](/time-zone-functions#convert_local_timestamp_to_utc) | Time Zone Functions | The function converts a timestamp in a specified local time zone to the UTC time zone. | | [CONVERT\_UTC\_TIMESTAMP\_TO\_LOCAL](/time-zone-functions#convert_utc_timestamp_to_local) | Time Zone Functions | The function converts a timestamp from the UTC time zone to a specified local time zone. | | [CAST\_TO\_TUPLE](/tuple-functions-and-operators) | Tuple Functions | Converts a tuple into another tuple of a different type. | | [CHAR](/tuple-functions-and-operators) | Tuple Functions | Converts a tuple to its string representation. | | [STRING\_TO\_TUPLE](/tuple-functions-and-operators) | Tuple Functions | Converts the string representation of a tuple (e.g 'tuple\<\>(1,2,NULL)') into a tuple. | | [TUPLE()](/tuple-functions-and-operators) | Tuple Functions | Construct a tuple of specified elements. Types of the tuple are inferred from the inner elements. | | [TUPLE\<\<>>](/tuple-functions-and-operators) | Tuple Functions | Construct a tuple of specified elements. NULL is also supported as an element. | | [CUME\_DIST](/window-aggregate-functions#cume_dist) | Window Aggregate Functions | Returns a number 0 \< n 1 and can be used to calculate the percentage of values less than or equal to the current value in the group. | | [DELTA](/window-aggregate-functions#delta) | Window Aggregate Functions | Computes the finite difference between successive values of expression under the specified ordering. This is a backwards difference, which means that, at degree one, the value for a given row is the difference between the value of expression for that row and the previous row. | | [DENSE\_RANK](/window-aggregate-functions#dense_rank) | Window Aggregate Functions | Assigns a number to each row in the result set with equal values having the same number. There will be no gaps between ranks. | | [DERIVATIVE](/window-aggregate-functions#derivative) | Window Aggregate Functions | Computes the difference quotient between successive values of expression with respect to expression2. | | [FIRST\_VALUE](/window-aggregate-functions#first_value) | Window Aggregate Functions | Returns the first value in the ordered result set. | | [LAG](/window-aggregate-functions#lag) | Window Aggregate Functions | Returns the row, which is the specified number backward from the current row. Default is 1 if offset is omitted. | | [LAST\_VALUE](/window-aggregate-functions#last_value) | Window Aggregate Functions | Returns the last value in the ordered result set. | | [LEAD](/window-aggregate-functions#lead) | Window Aggregate Functions | Returns the row, which is the specified number forward from the current row. Default is 1 if offset is omitted. | | [NTH\_VALUE](/window-aggregate-functions#nth_value) | Window Aggregate Functions | Returns the nth value in the ordered result set. | | [PERCENT\_RANK](/window-aggregate-functions#percent_rank) | Window Aggregate Functions | The value returned is 0 \< n ≤ 1 and can be used to calculate the percentage of values less than the current group, excluding the highest value. | | [PERCENTILE](/window-aggregate-functions#percentile) | Window Aggregate Functions | Returns the value that corresponds to the specified percentile (0 ≤ n ≤ 1) within the group. | | [RANK](/window-aggregate-functions#rank) | Window Aggregate Functions | Assigns a number to each row in the result set with equal values having the same number. There can be gaps between ranks. | | [RATIO\_TO\_REPORT](/window-aggregate-functions#ratio_to_report) | Window Aggregate Functions | Computes the ratio of a value to the sum of the set of values. | | [ROW\_NUMBER](/window-aggregate-functions#row_number) | Window Aggregate Functions | Assigns a unique number to each row in the result set. | | [ZSCORE](/window-aggregate-functions#zscore) | Window Aggregate Functions | Zscore of the sample based on the `stddev()` function. | | [ZSCOREP](/window-aggregate-functions#zscorep) | Window Aggregate Functions | Zscore of the sample based on the `stddevp()` function. | # Character and Binary Functions Source: https://docs.ocient.com/character-and-binary-functions Reference for Ocient SQL character and binary functions for string manipulation, encoding, decoding, padding, regular expressions, and byte-level operations. All string functions support Unicode characters. Functions that transform character case are locale-sensitive. These functions support the UTF-8 format: * RTRIM * LTRIM * LEFT * RIGHT * TRIM * TRANSLATE * RPAD * LPAD * SUBSTRING Index position values for character and string functions begin at position `1`. For example, the `"H"` in the string `"Hello"` is at position `1`. ### ASCII Returns the ASCII code value of the leftmost character of the character value. The ASCII function only supports ASCII characters in the input argument. **Syntax** ```sql SQL theme={null} ASCII(char) ``` **Example** ```sql SQL theme={null} SELECT ASCII('a'); ``` \*Output: \*`97` **Example** ```sql SQL theme={null} SELECT ASCII('bc'); ``` \*Output: \*`98` ### BIT\_LENGTH Returns the length of the character value in bits. **Syntax** ```sql SQL theme={null} BIT_LENGTH(char) ``` **Example** ```sql SQL theme={null} SELECT BIT_LENGTH('a'); ``` \*Output: \*`8` **Example** ```sql SQL theme={null} SELECT BIT_LENGTH('ab'); ``` \*Output: \*`16` **Example** ```sql SQL theme={null} SELECT BIT_LENGTH('ab4'); ``` \*Output: \*`24` ### BTRIM Alias for [TRIM](#trim). ### CHR Converts an integer value to a string. The value is first sign-extended to 8 bytes. Then, if it can be represented as 1 byte, a string is returned with that one byte. Otherwise, a string is returned of length 2 bytes, 4 bytes, or 8 bytes with the bytes set to the big-endian representation of the integer, depending on how many high order zero bytes there are in the integer. **Syntax** ```sql SQL theme={null} CHR(integer) ``` **Example** ```sql SQL theme={null} SELECT CHR(97); ``` \*Output: \*`a` ### CHAR\_LENGTH Alias for [LENGTH](#length). ### CHARACTER\_LENGTH Alias for [LENGTH](#length). ### CONCAT Concatenates two or more values, all of which must be binary, hash, or string data types. For strings, as long as one argument is a character value, the other arguments are implicitly cast to a character value. **Syntax** ```sql SQL theme={null} CONCAT(value1, value2 [, ...]) ``` | **Arguments** | **Data** **Type** | **Description** | | ------------------------ | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `value1, value2 [, ...]` | `BINARY`, `HASH`, or `CHAR` | Two or more values to concatenate.
If any argument is a character value, the others are also cast to a character value. | **Example** ```sql SQL theme={null} SELECT CONCAT('ocient',' ','data', ' ', 'warehouse'); ``` \*Output: \*`ocient data warehouse` **Example** ```sql SQL theme={null} SELECT 'ocient'||' data warehouse'; ``` \*Output: \*`ocient data warehouse` ### ENDSWITH Returns `true` if `x` ends with `y` and `false` otherwise. **Syntax** ```sql SQL theme={null} ENDSWITH(char1, char2) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | --------------------------------------------------------------------- | | `char1` | `CHAR` | A string to evaluate for whether it ends with `char2`. | | `char2` | `CHAR` | A string to evaluate for whether `char1` ends with it as a substring. | **Example** ```sql SQL theme={null} SELECT ENDSWITH('ocient data warehouse','warehouse'); ``` \*Output: \*`true` **Example** ```sql SQL theme={null} SELECT ENDSWITH('ocient data warehouse','db'); ``` \*Output: \*`false` **Example** ```sql SQL theme={null} SELECT ENDSWITH('tamaño','o'); ``` \*Output: \*`true` ### INITCAP For each word in the specified string, capitalize the first character if it is alphabetic. The system defines words as alphanumeric strings separated by non-alphanumeric characters. The system converts all other alphabetic characters to lowercase. **Syntaxes** ```sql SQL theme={null} INITCAP(char_string) INITCAP(char_string, delimiter) ``` | **Argument** | **Data** **Type** | **Description** | | ------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `char_string` | `CHAR` | A string to evaluate for initial capitalization. | | `delimiter` | `CHAR` | Optional.
One or more non-alphanumeric characters that specify the delimiter between words. The characters include all the ASCII printable non-alphanumeric characters except backquotes `` ` `` and equals `=`. Non-ASCII characters are not included.
ℹ️If you specify alphabetic characters for the delimiter, the function converts them to lowercase.
**Example**
`sql SQL
SELECT INITCAP('ocient');
`
\*Output: \*`Ocient` | ### INSTR Returns the index position of the first occurrence where the character value `char_substring` appears in the character value `char` by ignoring the case. **Syntax** ```sql SQL theme={null} INSTR(char, char_substring) ``` | **Argument** | **Data** **Type** | **Description** | | ---------------- | ----------------- | --------------------------------------------------------------------------------- | | `char` | `CHAR` | A string to evaluate for the first index position where `char_substring` appears. | | `char_substring` | `CHAR` | A string to evaluate for where it first appears in `char`. | **Example** ```sql SQL theme={null} SELECT INSTR('ocient dw dw', 'ocient'); ``` \*Output: \*`1` ```sql SQL theme={null} SELECT INSTR('ocient dw dw', 'dw'); ``` \*Output: \*`8` ### JSON\_EXTRACT\_PATH\_TEXT Returns the value for the key-value pair referenced by a series of path elements in a JSON string. **Syntax** ```sql SQL theme={null} JSON_EXTRACT_PATH_TEXT ( json_string, path [, path2 [, ...] ] [, null_if_invalid ] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `json_string` | `CHAR` | A JSON string. | | `path [, path2 [, ...] ]` | `CHAR` | One or more path elements in a JSON string.
You must include at least one path element up to a maximum of five, meaning the function can extract from paths nested up to five levels deep in the JSON string.
Path elements are case-sensitive. If a specified path element does not exist in the JSON string, the function returns `NULL`. | | `null_if_invalid` | `BOOLEAN` | Optional.
If you set this value to `TRUE`, the function returns `NULL` if the JSON string is invalid.
If you set this value to `FALSE`, the function returns an error when the JSON string is invalid.
The default value is `FALSE`. | **Examples** **Extract Values from Nested Paths** This example extracts the value based on two specified paths in the JSON string, `n4` and `n6`. ```sql SQL theme={null} SELECT JSON_EXTRACT_PATH_TEXT( '{"n2":{"n3":1},"n4":{"n5":99,"n6":"circle"}}', 'n4', 'n6' ); ``` \*Output: \*`"circle"` **Return NULL from Invalid JSON** In this example, the query requests the same nested paths, but the JSON is invalid because all of the keys lack quotation marks. The query returns `NULL` because the `null_if_invalid` argument equals `TRUE`. ```sql SQL theme={null} SELECT JSON_EXTRACT_PATH_TEXT( '{n2:{n3:1},n4:{n5:99,n6:"circle"}}', 'n4', 'n6', TRUE ); ``` \*Output: \*`NULL` ### LCASE Alias for [LOWER](#lower). ### LEFT Return the number of characters in the string equal of the value `integer`. If `integer` is negative, the function returns all but the last `integer` characters. **Syntax** ```sql SQL theme={null} LEFT(char, integer) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `char` | `CHAR` | A string to be modified by returning the number of characters equal to `integer`. | | `integer` | `INT` | The number of characters to return from the `char` string.

If `integer` is negative, the function returns all but the last `integer` characters. | **Example** ```sql SQL theme={null} SELECT LEFT('ocient data warehouse', 8); ``` \*Output: \*`ocient d` **Example** ```sql SQL theme={null} SELECT LEFT('ocient data warehouse', -3); ``` \*Output: \*`ocient data wareho` ### LENGTH Alias for CHAR\_LENGTH and CHARACTER\_LENGTH. Returns the length of the value. For character data types, this value is in terms of characters. For binary data types, this value is in terms of bytes. **Syntax** ```sql SQL theme={null} LENGTH(character_or_binary_value) ``` **Example** ```sql SQL theme={null} SELECT LENGTH('ocient data warehouse'); ``` \*Output: \*`21` ### LOCATE Alias for POSITION. Returns the index position of the first occurrence of the character value `substring` in the character value `string`. Optionally, you can also include an additional value `offset` to offset the LOCATE function by the specified number of spaces. Index positions begin at `1`. **Syntax** ```sql SQL theme={null} LOCATE(substring, string [, offset] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `substring` | `CHAR` | A substring to be found for its first position in string. | | `string` | `CHAR` | A string to be evaluated for the first position of `substring`. | | `offset` | `INT` | Optional. A number of index positions in `string` to offset the search for the `substring` value.
This index position must be a positive integer, and it starts from the left of `string`. | **Example** ```sql SQL theme={null} SELECT LOCATE('ware', 'ocient data warehouse'); ``` \*Output: \*`13` **Example** ```sql SQL theme={null} SELECT LOCATE('e', 'ocient'); ``` \*Output: \*`4` **Example** In this example, the index starts at position 5. This means the function skips the initial `'e'` in the string. Instead, it returns the second `e` at index position 15. ```sql SQL theme={null} SELECT LOCATE('e', 'ocient database', 5); ``` \*Output: \*`15` ### LOWER Alias for LCASE. Convert string to lowercase. **Syntax** ```sql SQL theme={null} LOWER(char) ``` **Example** ```sql SQL theme={null} SELECT LOWER('Ocient'); ``` \*Output: \*`ocient` **Example** ```sql SQL theme={null} SELECT LOWER('OCIENT'); ``` \*Output: \*`ocient` ### LPAD Pad the input text to the specified length with the pad string on the left side. If text is longer than length, it is truncated to `length` characters. If the argument pad is not provided, the space character is used. **Syntax** ```sql SQL theme={null} LPAD(string, length [, pad_character] ) ``` | **Argument** | **Data** **Type** | **Description** | | --------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | A string to be padded on its left side with additional characters, so that it equals the `length` value. | | `length` | `INT` | The number of characters for `string` to total.
If the `string` is already longer than `length`, the function truncates `string` to equal the `length` value. | | `pad_character` | `CHAR` | Optional. A string of one or more characters to use to pad `string` to equal the `length` value.
If not provided, `pad_character` uses a whitespace character to pad. | **Example** ```sql SQL theme={null} SELECT LPAD('ocient data warehouse',30,'ab'); ``` \*Output: \*`ababababaocient data warehouse` **Example** ```sql SQL theme={null} SELECT LPAD('ocient data warehouse',6); ``` \*Output: \*`ocient` ### LTRIM Removes leading blanks from the string value `string`. Alternatively, you can specify a second string value `trim_character`. If you specify the `trim_character` value, the LTRIM function removes all leading instances of the `trim_character` value from the `string`. **Syntax** ```sql SQL theme={null} LTRIM(string [, trim_character] ) ``` | **Argument** | **Data** **Type** | **Description** | | ---------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | A string to be trimmed on its left side. | | `trim_character` | `CHAR` | Optional. A string of one or more characters to be trimmed from the left side of `string`. Each character is trimmed individually, not as a cohesive substring.
If you do not specify this argument, this value defaults to removing all whitespace from the left side of `string`. | **Example** ```sql SQL theme={null} SELECT LTRIM(' ocient'); ``` \*Output: \*`ocient` **Example** ```sql SQL theme={null} SELECT LTRIM('aaaaaocient','a'); ``` \*Output: \*`ocient` **Example** In this example, all characters `'abeo '` are removed from the left side of the string. ```sql SQL theme={null} SELECT LTRIM('aaeabe ocient','abeo '); ``` \*Output: \*`cient` ### MD5 Returns the hexadecimal string (all lowercase) representing the md5 hash of `char`. **Syntax** ```sql SQL theme={null} MD5(char) ``` **Example** ```sql SQL theme={null} SELECT md5('ocient'); ``` \*Output: \*`438f03cf6e9ddf8793e02db25f2d2f88` ### MID Alias for [SUBSTRING](#substring). ### OCTET\_LENGTH Returns the length in bytes of a character or binary value. **Syntax** ```sql SQL theme={null} OCTET_LENGTH(value) ``` **Example** ```sql SQL theme={null} SELECT OCTET_LENGTH('a'); ``` \*Output: \*`1` **Example** ```sql SQL theme={null} SELECT OCTET_LENGTH('ocient'); ``` \*Output: \*`6` ### POSITION Alias for [LOCATE](#locate). ### REGEXP\_COUNT Searches a string for all occurrences of a regular expression pattern. The function returns an integer representing the number of times the regular expression pattern occurs in the string. **Syntax** ```sql SQL theme={null} REGEXP_COUNT( string, pattern [, position [, parameters ] ] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `string` | `CHAR` | The string to search, specified as a character string. | | `pattern` | `CHAR` | The regular expression pattern to use in the search, specified as a character string. | | `position` | `INT` | Optional.
The position, specified as a positive integer, that represents the position within the `string` string where to begin the search, based on the number of characters.
If this integer exceeds the number of characters in `string`, then the function returns `0`.
The default value is `1`. | | `parameters` | `CHAR` | Optional.
Parameters, specified as a character string, that contains one or more characters representing regular expression options for pattern matching. Supported options are:
`c` — Perform case-sensitive matching. The System enables this type of matching by default.
`i` — Perform case-insensitive matching.
`p` — Interpret the pattern using the Perl Compatible Regular Expression (PCRE) dialect. For details, see [PCRE](https://www.boost.org/doc/libs/1_33_1/libs/regex/doc/syntax_perl.html#Perl). The Ocient System enables this interpretation by default. | **Example** This example searches the `'ABABDaSGRESaB'` string for the count of occurrences of the `'AB'` string by ignoring the case sensitivity. The search starts at position `1`. ```sql SQL theme={null} SELECT REGEXP_COUNT('ABABDaSGRESaB', 'AB', 1, 'i'); ``` \*Output: \*`3` ### REGEXP\_INSTR Searches a string using a regular expression pattern and returns an integer representing the start position or end position of the substring that matches. The function returns `0` if no match is found. The `REGEXP_INSTR` function is similar to the [POSITION](#position) function, but it allows greater precision with regular expressions. **Syntax** ```sql SQL theme={null} REGEXP_INSTR( source_string, pattern [, position [, occurrence] [, option [, parameters ] ] ] ] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | The string to search, specified as a character string.
Note that SQL escape sequences for string literals can override regular expression escape sequences. For details, see [String Literals and Escape Sequences](/data-query-language-dql-statement-reference#string-literals-and-escape-sequences). | | `pattern` | `CHAR` | The regular expression pattern to use in the search, specified as a character string. | | `position` | `INT` | Optional.
A positive integer that represents the position within the `string` string where to begin the search, based on the number of characters. This value alters only the start position for the match, not the returned string position.
If this integer exceeds the number of characters in `string`, then the function returns `0`.
The default value is `1`. | | `occurrence` | `INT` | Optional.
A positive integer that represents the occurrence of a positive pattern match to return.
The default value is `1`, which means the function returns the first substring that matches the regular expression pattern. | | `option` | `INT` | Optional.
An integer that specifies whether to return the position of the matching start position or the end position. Your choices are:
\* `0`: Returns the start position of the match.
\* `1`: Returns the end position of the match `+1`.
The function treats any nonzero integer value as `1`.
The default value is `0`. | | `parameters` | `CHAR` | Optional.
A string containing one or more characters that represents the regular expression options for pattern matching. Supported options are:
`c` — Perform case-sensitive matching. The Ocient System enables this type of matching by default.
`i` — Perform case-insensitive matching.
`e` — Extract the substring using a regular expression subpattern. This subpattern is enclosed in parentheses in the regular expression.
The function uses the full regular expression pattern for the match but returns only the first subpattern match.
`p` — Interpret the pattern using the Perl Compatible Regular Expression (PCRE) dialect. For details, see [PCRE](https://www.boost.org/doc/libs/1_33_1/libs/regex/doc/syntax_perl.html#Perl). The Ocient System enables this interpretation by default. | **Example** The query searches the website URL for the substring preceded by the `#` character. The function includes optional arguments to start the search at position `9` and match the first occurrence. The last specified optional argument directs the function to return the ending position of the matching substring. ```sql SQL theme={null} SELECT REGEXP_INSTR( 'https://docs.ocient.com/character-binary-functions#J28jB', '#\w+', 9, 1, 1 ); ``` \*Output: \*`57` ### REGEXP\_REPLACE Searches a string for all occurrences of a regular expression pattern. The function replaces every match occurrence of the pattern with a new string. The `REGEXP_REPLACE` function is similar to the [REPLACE](#replace) and [TRANSLATE](#translate) functions, but it allows greater precision with regular expressions. **Syntax** ```sql SQL theme={null} REGEXP_REPLACE( string, pattern [, replace_string [ , position [, parameters ] ] ] ) ``` | **Argument** | **Data** **Type** | **Description** | | ---------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | The string to search, specified as a character string.
Note that SQL escape sequences for string literals can override regular expression escape sequences. For details, see [String Literals and Escape Sequences](/data-query-language-dql-statement-reference). | | `pattern` | `CHAR` | The regular expression pattern to use in the search, specified as a character string. | | `replace_string` | `CHAR` | Optional.
A string to replace all occurrences of the regular expression pattern in `string`.
This string can include references to capture groups in the regular expression `pattern`. To reference capture groups, use a `$` followed by the group number. For example, `$1` references the first capture group, and `$2` references the second.
The default value is an empty string `''`. | | `position` | `INT` | Optional.
A positive integer that represents the position within the string to begin the search, based on the number of characters.
If this integer exceeds the number of characters in `string`, then the function returns the original `string`.
The default value is `1`. | | `parameters` | `CHAR` | Optional.
A string containing one or more characters representing regular expression options for pattern matching. Supported options are:
`c` — Perform case-sensitive matching. The Ocient System enables this type of matching by default.
`i` — Perform case-insensitive matching.
`p` — Interpret the pattern using the Perl Compatible Regular Expression (PCRE) dialect. For details, see [PCRE](https://www.boost.org/doc/libs/1_33_1/libs/regex/doc/syntax_perl.html#Perl). The Ocient System enables this interpretation by default. | **Examples** **Replace Text Using a Substring** The query replaces the matching substring `'#J28jB'` with the characters `'#0FqD_'`. The search begins at position `9`. ```sql SQL theme={null} SELECT REGEXP_REPLACE( 'https://docs.ocient.com/character-binary-functions#J28jB', '#\w+', '#0FqD_', 9 ); ``` \*Output: \*`/character-and-binary-functions` **Replace Text Using Multiple Capture Groups** This example uses three capture groups to take an unformatted phone number and convert it into the `(XXX) XXX-XXXX` format. The `replace_string` argument references each capture group as `$1`, `$2`, and `$3`. ```sql SQL theme={null} SELECT REGEXP_REPLACE( '5558675309', '(\d{3})(\d{3})(\d{4})', '($1) $2-$3' ); ``` \*Output: \*`(555) 867-5309` ### REGEXP\_SUBSTR Returns one substring from a string that matches a specified regular expression pattern. `REGEXP_SUBSTR` is similar to the [SUBSTR](#substr) function, but it allows greater precision with regular expressions. If the pattern produces no matches, the function returns an empty string. **Syntax** ```sql SQL theme={null} REGEXP_SUBSTR( string, pattern [, position [, occurrence [, parameters ] ] ] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | The string to search, specified as a character string.
Note that SQL escape sequences for string literals can override regular expression escape sequences. For details, see [String Literals and Escape Sequences](/data-query-language-dql-statement-reference). | | `pattern` | `CHAR` | The regular expression pattern to use in the search, specified as a character string. | | `position` | `INT` | Optional.
A positive integer that represents the position within the string to begin the search, based on the number of characters.
If this integer is greater than the number of characters in `string`, then the function returns the original `string`.
The default value is `1`. | | `occurrence` | `INT` | Optional.
A positive integer that represents the occurrence of a positive pattern match to return.
If this value exceeds the number of matches, then the function returns NULL.
The default value is `1`, which means the function returns the first substring that matches the regular expression pattern. | | `parameters` | `CHAR` | Optional.
A string containing one or more characters representing regular expression options for pattern matching. Supported options are:
`c` — Perform case-sensitive matching. The Ocient System enables this type of matching by default.
`i` — Perform case-insensitive matching.
`e` — Extract the substring using a regular expression subpattern. This subpattern is enclosed in parentheses in the regular expression.
The function uses the full regular expression pattern for the match, but it returns only the first subpattern match.
If there is no subexpression in the `pattern` argument, then the `e` parameter is ignored.
`p` — Interpret the pattern using the Perl Compatible Regular Expression (PCRE) dialect. For details, see [PCRE](https://www.pcre.org/). The Ocient System enables this interpretation by default. | **Example** The query returns the first occurrence of the `#J28jB` string by using the regular expression pattern `'#\w+'`. The search starts at position `9`. ```sql SQL theme={null} SELECT REGEXP_SUBSTR( 'https://docs.ocient.com/character-binary-functions#J28jB', '#\w+', 9, 1 ); ``` \*Output: \*`#J28jB` ### REPEAT Repeats the `char` string `num` times without spaces. **Syntax** ```sql SQL theme={null} REPEAT(char, num) ``` **Example** Repeat the `a` string five times. ```sql SQL theme={null} SELECT REPEAT('a', 5); ``` \*Output: \*`aaaaa` ### REPLACE Replaces all occurrences of `substr_to_remove` in the character value `string` with `substr_to_replace`. **Syntax** ```sql SQL theme={null} REPLACE(string, substr_to_remove, substr_to_replace) ``` | **Argument** | **Data** **Type** | **Description** | | ------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `string` | `CHAR` | A string to alter by replacing all instances of `substr_to_remove` with `substr_to_replace`. | | `substr_to_remove` | `CHAR` | A substring to remove from `string`, replacing all instances with the `substr_to_replace` value. If `substr_to_remove` is the empty string, the system returns `string`. | | `substr_to_replace` | `CHAR` | A substring to replace all instances of `substr_to_remove`. | **Example** ```sql SQL theme={null} SELECT REPLACE('abcabcabcabc', 'ab', '$'); ``` \*Output: \*`$c$c$c$c` ### REVERSE Reverse the input string. **Syntax** ```sql SQL theme={null} REVERSE(char) ``` **Example** ```sql SQL theme={null} SELECT REVERSE('abcd'); ``` \*Output: \*`dcba` ### RIGHT Return the number of trailing characters in the string equal to the value `integer`. **Syntax** ```sql SQL theme={null} RIGHT(char, integer) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `char` | `CHAR` | A string to evaluate to return the number of trailing characters equal to `integer`. | | `integer` | `INT` | The number of characters of return from the end of the `char` string.
If `integer` is negative, the function returns all but the first `integer` characters. | **Example** ```sql SQL theme={null} SELECT RIGHT('ocient data warehouse', 8); ``` \*Output: \*`arehouse` **Example** ```sql SQL theme={null} SELECT RIGHT('ocient data warehouse', -3); ``` \*Output: \*`ent data warehouse` ### RPAD Pad the input text to the specified length with the pad string on the right side. If text is longer than length, it is truncated to `length` characters. If the argument pad is not provided, the space character is used. **Syntax** ```sql SQL theme={null} RPAD(character_value_text, integral_value_length [, character_value_pad] ) ``` | **Argument** | **Data** **Type** | **Description** | | --------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | A string to be padded on its right side with additional characters, so that it equals the `length` value. | | `length` | `INT` | The number of characters for `string` to total.
If the `string` is already longer than `length`, the function truncates `string` to equal the `length` value. | | `pad_character` | `CHAR` | Optional. A string of one or more characters to use to pad `string` to equal the `length` value.
If not provided, `pad_character` uses a whitespace character to pad. | **Example** ```sql SQL theme={null} SELECT RPAD('ocient data warehouse',30); ``` \*Output: \*`ocient data warehouse ` **Example** ```sql SQL theme={null} SELECT RPAD('ocient data warehouse',30,'ab'); ``` \*Output: \*`ocient data warehouseababababa` ### RSUBSTRING Returns the substring from the right side of a string, based on a specified length. **Syntax** ```sql SQL theme={null} RSUBSTRING(string, integer) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | A string to be evaluated to return a substring based on the `integer` value. | | `integer` | `INT` | The number of characters to return from the right side of the `string` value.
This value must be positive. | **Example** ```sql SQL theme={null} SELECT RSUBSTRING('ocient data warehouse',14); ``` \*Output: \*`data warehouse` ### RTRIM Removes trailing blanks from the string value `string`. Alternatively, you can specify a second string value `trim_character`. If you specify the `trim_character` value, the RTRIM function removes all trailing instances of the `trim_character` value from the `string`. **Syntax** ```sql SQL theme={null} RTRIM(string [, trim_character] ) ``` | **Argument** | **Data** **Type** | **Description** | | ---------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `string` | `CHAR` | A string to be trimmed on its right side. | | `trim_character` | `CHAR` | Optional. A string of one or more characters to be trimmed from the right side of `string`. Each character is trimmed individually, not as a cohesive substring.
If you do not specify this argument, this value defaults to removing all whitespace from the right side of `string`. | **Example** ```sql SQL theme={null} SELECT RTRIM('ocient '); ``` \*Output: \*`ocient` **Example** ```sql SQL theme={null} SELECT RTRIM('ocientaaaaaa','a'); ``` \*Output: \*`ocient` ### SHA1 Uses the \[SHA-1]\([https://en.wikipedia.org/wiki/SHA-1#:\~:text=In%20cryptography%2C%20SHA%2D1%20(,rendered%20as%2040%20hexadecimal%20digits](https://en.wikipedia.org/wiki/SHA-1#:~:text=In%20cryptography%2C%20SHA%2D1%20\(,rendered%20as%2040%20hexadecimal%20digits).) cryptographic hash function to convert a string into a 40-character string representing the hexadecimal value of a 160-bit checksum. **Syntax** ```sql SQL theme={null} SHA1(string) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ---------------------------------------------------------------- | | `string` | `CHAR` | A string to convert using the SHA-1 cryptographic hash function. | **Example** ```sql SQL theme={null} SELECT SHA1('Ocient'); ``` \*Output: \*`0c2a9a042b9f047f875c3414e7a4f4c53efbe082` ### SPACE Returns a string of repeated spaces equal to the number value, `repeat`. You can join this to another string by using the [CONCAT](#concat) function. **Syntax** ```sql SQL theme={null} SPACE(repeat) ``` **Example** ```sql SQL theme={null} SELECT CONCAT(SPACE(10),'end'); ``` \*Output: \*` end` ### SPLIT\_PART Split the value `string` based on the `delimiter` value. The function returns a substring from the split operation based on the `index` value (starting from 1). **Syntax** ```sql SQL theme={null} SPLIT_PART(string, delimiter, index) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ----------------------------------------------------------------------------- | | `string` | `CHAR` | A string to be split based on the `delimiter` value. | | `delimiter` | `CHAR` | The character to use as a delimiter in `string`. | | `index` | `INT` | The index of the substring to return. The first index position starts at `1`. | **Example** ```sql SQL theme={null} SELECT SPLIT_PART('id|name|address','|',1); ``` \*Output: \*`id` **Example** ```sql SQL theme={null} SELECT SPLIT_PART('id|name|address','|',2); ``` \*Output: \*`name` **Example** ```sql SQL theme={null} SELECT SPLIT_PART('id,name,address',',',3); ``` \*Output: \*`address` ### SPLIT\_TO\_ARRAY Splits a string into an array of substrings. **Syntax** ```sql SQL theme={null} SPLIT_TO_ARRAY(string, [ delimiter ]) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------ | | `string` | `CHAR` | A string to split into an array of substrings. | | `delimiter` | `CHAR` | Optional.
The delimiter that divides the string.
The default value is a comma (`,`). | **Example** This example specifies `|` as the delimiter to split the input string. ```sql SQL theme={null} SELECT SPLIT_TO_ARRAY('AB|CD|EF', '|'); ``` \*Output: \*`['AB','CD','EF']` ### STARTSWITH Returns `true` if `string` starts with `substring` and `false` otherwise. **Syntax** ```sql SQL theme={null} STARTSWITH(string, substring) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------- | | `string` | `CHAR` | A string to search. | | `substring` | `CHAR` | A substring to find at the beginning of `string`. | **Example** ```sql SQL theme={null} SELECT STARTSWITH('ocient','o'); ``` \*Output: \*`true` **Example** ```sql SQL theme={null} SELECT STARTSWITH('ocient','c'); ``` \*Output: \*`false` ### STRPOS Equivalent to using [LOCATE](#locate) as `LOCATE(substring, string)`. Note the reversed argument order. **Syntax** ```sql SQL theme={null} STRPOS(string, substring) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | --------------------------------------------------------------- | | `string` | `CHAR` | A string to be evaluated for the first position of `substring`. | | `substring` | `CHAR` | A substring to be found for its first position in `string`. | **Example** ```sql SQL theme={null} SELECT STRPOS('ocient data warehouse','house'); ``` \*Output: \*`17` ### SUBSTR Alias for [SUBSTRING](#substring). ### SUBSTRING Alias for SUBSTR and MID. Returns the substring of a character or binary value that starts with the position specified by the second argument and that ends with the position specified by one less than the sum of the second and third arguments. When the sum of the second and third arguments is less than two, the function returns the empty string. **Syntax** ```sql SQL theme={null} SUBSTRING(string, start_position [, length] ) ``` | **Argument** | **Data** **Type** | **Description** | | ---------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | A string to be truncated based on the `start_position` and `end_position` values. | | `start_position` | `INT` | The starting position to return a substring. The first index position starts at `1`. | | `length` | `INT` | Optional. The number of characters from the `start_position` to include in the returned substring.
If unspecified, SUBSTRING returns all characters after the `start_position`. | **Example** ```sql SQL theme={null} SELECT SUBSTRING('ocient data warehouse',8); ``` \*Output: \*`data warehouse` **Example** ```sql SQL theme={null} SELECT SUBSTRING('ocient data warehouse',8,4); ``` \*Output: \*`data` ### TO\_CHAR Converts a numeric, date, or timestamp value into a `CHAR` date type. ```sql SQL theme={null} TO_CHAR(value, format) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `value` | `DATE`, `TIMESTAMP`, or all numeric types | A numeric, date, or timestamp value to be converted into a `CHAR` date type. | | `format` | `CHAR` | The format used for the `CHAR` conversion.
For information on data type formats, see the [Formatting Functions](/formatting-functions) page. | **Example** ```sql SQL theme={null} SELECT TO_CHAR(20200610132514/1000000,'9999-99-99'); ``` \*Output: \*`2020-06-10` **Example** ```sql SQL theme={null} SELECT TO_CHAR(20200610132514%1000000,'99:99:99'); ``` \*Output: \*`13:25:14` ### TRANSLATE Replaces specified characters in a provided string with a separate set of characters. Characters specified in the `char_to_remove` set are replaced with characters in the `char_to_replace` set based on the corresponding index position. **Syntax** ```sql SQL theme={null} TRANSLATE(string, char_to_remove, char_to_replace) ``` | **Argument** | **Data** **Type** | **Description** | | ----------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | A string to have specific characters transposed based on the `char_to_remove` and `char_to_replace` values. | | `char_to_remove` | `CHAR` | The characters to be removed from `string`.
Characters specified in the `char_to_remove` set are replaced with characters in the `char_to_replace` set based on the corresponding index position.
If `char_to_remove` is longer than `char_to_replace`, occurrences of the extra characters are removed. | | `char_to_replace` | `CHAR` | The characters to replace the removed characters in `char_to_remove`. | **Example** ```sql SQL theme={null} SELECT TRANSLATE('abcdef','ace','ghi'); ``` \*Output: \*`gbhdif` ### TRIM Alias for BTRIM. Trim leading and trailing space characters from the string. Alternatively, you can specify a second string value `trim_char`. If a `trim_char` value is provided, the `TRIM` function removes all leading and trailing instances of the `trim_char` value from the string. **Syntax** ```sql SQL theme={null} TRIM(string [, trim_char] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `string` | `CHAR` | A string to be trimmed of leading and trailing space characters, or any other characters specified by the `trim_char` argument. | | `trim_char` | `CHAR` | Optional. If specified, this removes an alternative substring from the start or end of the `string` value. | **Example** ```sql SQL theme={null} SELECT TRIM(' trimmed string '); ``` \*Output: \*`trimmed string` **Example** ```sql SQL theme={null} SELECT TRIM('aaaaaaaatrimmed stringaaaaaaaaa', 'a'); ``` \*Output: \*`trimmed string` ### UCASE Alias for [UPPER](#upper). ### UPPER Alias for UCASE. Convert string to upper case. **Syntax** ```sql SQL theme={null} UPPER(character_value) ``` **Example** ```sql SQL theme={null} SELECT UPPER('ocient'); ``` \*Output: \*`OCIENT` ### Concatenate Operator(`||`) The `||` operator concatenates two strings. If you specify a NULL string, the result is NULL. `||` **Syntax** ```sql SQL theme={null} string1 || string2 ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ---------------------------------------------- | | `string1` | `CHAR` | A string to concatenate. | | `string2` | `CHAR` | A string to concatenate with the first string. | **Examples** **Concatenate Two Strings** Concatenate two strings. ```sql SQL theme={null} SELECT 'hello ' || 'world'; ``` Output: `'hello world'` **Concatenate a NULL String** Concatenate two strings, one of which is NULL. ```sql SQL theme={null} SELECT NULL || 'world'; ``` Output: `NULL` ## Related Links [WHERE](/data-query-language-dql-statement-reference#where) [Formatting Functions](/formatting-functions) # Checking System After Configuration Source: https://docs.ocient.com/checking-system-after-configuration Verify that an Ocient System is healthy after configuration by checking node status, storage spaces, network connectivity, and core service operation. The `postcheck` command helps capture critical node health states and metrics after the configuration process. You can run the command as shown in this example. The flags included after the `precheck` command are optional. See [Flags for Command Execution](#flags-for-command-execution) for descriptions. ```shell Shell theme={null} $ sudo /usr/local/bin/postcheck --dsn ocient://:@:/ ``` This table describes each placeholder in the DSN: | **Placeholder** | **Description** | | --------------- | ------------------------------------------------------------------------------ | | `` | The username for the Ocient database connection (for example, `admin@system`). | | `` | The password associated with the specified user. | | `` | The hostname or IP address of the SQL Node to connect to. | | `` | The port number on which the SQL Node is listening. | | `` | The name of the database to connect to. | The command invokes a script from the command line on operating systems supported by Ocient. The script sends output to standard output (STDOUT) and standard error (STDERR) with the option to send a text file or a JSON file. To specify the file option, use this flag: `--logfile-location ` where `` is the directory where the `postcheck` command creates the log file. Before you begin, ensure that all steps in the [Ocient System Bootstrapping](/ocient-system-bootstrapping) process are complete. ## Prerequisites The script requires and the Ocient JDBC JAR to run. In most cases, you should use this script on client servers rather than database nodes. To get the post-check script onto a client, perform the `scp` operation on the wheel onto the server and run `python3 -m pip install --no-index /postcheck*.whl`, and replace `` with the path to the directory of the wheel file on your server. ## Flags for Command Execution Use the `postcheck` command to run the script that checks the system configuration after you configure Ocient. You can execute the command using these flags. | **Flag Name** | **Required** | **Values** | **Default** | **Description** | **Example** | | ------------------ | ------------ | ----------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | --dsn | Yes | Many | Not applicable | Data source name (DSN). The script prompts for your credentials. | `--dsn ocient://host:port/path` connects you to the specified cluster where `host` is the host name, `port` is the port number, and `path` is the path on the host for the connection. | | --logfile-location | No | Non-empty string | /var/opt/ocient/log | Name of directory that contains the log file created from the execution of the script. | `--logfile-location /user/query/logs` generates a log file in the `/user/query/logs` directory. If this directory does not exist, the command issues a warning. | | --print-level | No | verbose or silent | verbose | Denotes the level of detail in the output. See the “Outputs” section. | `--print-level verbose` provides a detailed report with information about the checks that Ocient performs. `--print-level silent` provides the overall status from the check of the system. | | --timeout | No | Any integer | 3 | Denotes the default timeout (in seconds) for running any command-line command in the script. In most cases, the default is enough, so you do not need to specify this setting. | `--timeout 4` runs checks with a timeout of 4 seconds for each command. | | --jar-path | No | Non-empty string | /opt/ocient/current/ocient-jdbc4.jar | Denotes the path to the Ocient JDBC JAR. The default value is the usual location of the JAR file on an Ocient System. This path must be accurate for the postcheck command to run correctly. | `--jar-path /opt/ocient/test_dir/jdbc.jar` uses the JAR at `/opt/ocient/test_dir/jdbc.jar` to run JDBC commands for the `postcheck` command. | | --json | No | Not applicable | False | If you specify the --json flag, the command prints output in JSON format to STDOUT. If you do not specify this flag, the command prints human-readable output to STDOUT. | `--json` prints output in JSON format to STDOUT. | ## Command Output The command has two types of output, silent or verbose, which you can specify by using the `--print-level` flag at the command line. **Silent** The `silent` value for the `--print-level` flag only produces a status code. The code is 0 for success, 1 for error, and 2 for warning. **Verbose** The `verbose` value for the `--print-level` flag prints either human-readable output by default or output in JSON format (if you specify the `--json` flag). The output contains all status details. This example shows human-readable output. ```Text Text theme={null} Warnings: - Foundation Nodes storage are non-uniform. OS Configuration - OK ================ Huge Page Settings - OK isolcpus settings - OK Memory - OK ====== Foundation Nodes: 1024 GB SQL Nodes: 1024 GB Loader Nodes: 1536 GB Total DRAM: 123120 GB ... ``` This example shows output in JSON format. ```json JSON theme={null} {"status": 2, "components": { "firmware": { "status": 0, "message": "SUCCESS: NVMe drive firmware is consistent across nodes." } "symmetry": { "status": 2, "message": "WARNING: non-uniform configuration." "storage": { "status": 2, "foundation": { "status": 0, "message": "SUCCESS: storage consistent across Foundation Nodes." "foundation1": 16TB, ... } ... } } ``` ## Related Links [Ocient System Bootstrapping](/ocient-system-bootstrapping) [Ocient Application Configuration](/ocient-application-configuration) # Checking System Configuration Before Bootstrapping Source: https://docs.ocient.com/checking-system-configuration-before-bootstrapping Validate Ocient System configuration before bootstrapping nodes, including hostnames, networking, disk layout, and configuration files to prevent setup errors. The `precheck` command helps capture critical node health states and metrics before bootstrapping your installation. You can run the command as shown in this example. The flags included after the `precheck` command are optional. See [Optional Flags for Command Execution](#optional-flags-for-command-execution) for descriptions. ```shell Shell theme={null} $ sudo precheck --print-level verbose --node-type metadata --logfile-location /home/admin ``` The command invokes a script from the command line on operating systems supported by Ocient. The script sends output to standard output (STDOUT) and standard error (STDERR). To specify the file option, use this flag: `--logfile-location ` where `` is the directory where the `precheck` command creates the log file. Critically, this script captures information about huge page configuration. ## Optional Flags for Command Execution Use the `precheck` command to run the script that checks the system configuration. You can execute the command using the optional flags shown in this table. | **Flag Name** | **Values** | **Default** | **Description** | **Example** | | ----------------------- | -------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | --logfile-location | Non-empty string | `/var/opt/ocient/log` | Name of the directory that contains the log file created from the execution of the script. | `--logfile-location /user/query/logs` generates a log file in the `/user/query/logs` directory. If this directory does not exist, the command issues a warning. | | --print-level | verbose or silent | verbose | Denotes the level of detail in the output. See the “Outputs” section. | `--print-level verbose` provides a detailed report with information about the checks that Ocient performs. --print\_level silent provides the overall status from the check of the system. | | --node-type | metadata, lts, streamloader, sql, or unknown | unknown | Denotes the current node type. If you do not specify this setting, the script tries to infer node type based on metrics it collects. | `--node-type lts` runs checks with values and constraints that pertain to an `lts` node. | | --timeout | Any integer | 3 | Denotes the default timeout (in seconds) to run any command-line command in the script. In most cases, the default is enough, so you do not need to specify this setting. | `--timeout 4` runs checks with a timeout of 4 seconds for each command. | | --nvme-driver-util-path | Non-empty string | /opt/ocient/scripts/nvme-driver-util.sh | Path to `nvme-driver-util` script that the precheck script uses. The default value is the location of the script on most installations, so in most cases, you do not need to specify this setting. | `--nvme-driver-util-path /opt/ocient/test_dir/nvme-driver-util.sh` uses `/opt/ocient/test_dir/nvme-driver-util.sh` as the path to the utility script. | | --json | Not applicable | False | If you specify the `--json` flag, the command prints output in JSON format to STDOUT. If you do not specify this flag, the command prints human-readable output to STDOUT. | `--json` prints output in JSON format to STDOUT. | ## Command Output The command has two types of output, silent or verbose, which you can specify by using the `--print-level` flag at the command line. **Silent** The `silent` value for the `--print-level` flag only produces a status code. The code is 0 for success, 1 for error, and 2 for warning. **Verbose** The `verbose` value for the `--print-level` flag prints either human-readable output by default or output in JSON format (if you specify the `--json` flag). The output contains all status details. This example shows human-readable output. ```Text Text theme={null} Warnings: - isolcpus not configured - recommended for performance. - No eligible NVMe payload drives found for node. - bootstrap.conf file does not exist. Create a file at /var/opt/ocient/bootstrap.conf with the correct bootstrapping settings. If this is the initial node in the system set "initialSystem: true" in the bootstrap.conf file. If this is not the initial node, use the "adminHost" setting to specify the address of the initial node. To add custom settings, follow the instructions in Bootstrapping and Ocient System in the User Documentation. - SELinux is enabled. This might prevent Ocient from functioning properly. Please disable or set to permissive mode. Memory - OK ====== ramconfig - SUCCESS: RAM configuration is adequate. Firmware and Hardware - ERROR ==================== nonbootdrives - WARNING: No eligible NVMe payload drives found for node ... ``` This example shows output in JSON format. ```json JSON theme={null} {"status": 1, "components": { "enforce_status": { "status": 2, "message": "WARNING: SELinux is enabled. This might prevent Ocient from functioning properly. Please disable or set to permissive mode" } "bootstrap": { "status": 2, "message": "WARNING: bootstrap.conf file does not exist. Create a file at /var/opt/ocient/bootstrap.conf with the correct bootstrapping settings. If this is the initial node in the system set "initialSystem: true" in the bootstrap.conf file. If this is not the initial node, use the "adminHost" setting to specify the address of the initial node. To add custom settings, follow the instructions in Bootstrapping and Ocient System in the User Documentation." } ... } } ``` ## Related Links [Ocient System Bootstrapping](/ocient-system-bootstrapping) [Ocient Application Configuration](/ocient-application-configuration) # Classification Analysis Source: https://docs.ocient.com/classification-analysis Use classification analysis techniques within Ocient with OcientML to categorize data, helping in decision-making and pattern identification. Classification machine learning algorithms are available in . There are many different popular classification algorithms, which work in slightly different ways with different strengths and weaknesses. These tutorials and examples cover some basic rules for using classification algorithms and how you can take advantage of these models in . It can be very challenging to anticipate which classification model is best suited for a data set. Often, it is best to experiment with multiple different models to see what happens. ## Logistic Regression Despite its name, logistic regression is a classification algorithm, not a regression algorithm. Logistic regression is generally a good first step for trying out a classification model before moving on to more advanced options. Logistic regression shares many restrictions with the support vector machines model. Both models require that all features are numeric. However, you can still use non-numeric data if you convert it using one-hot encoding. The class labels can be any data type. Use one-hot encoding to convert non-numeric data into numeric form. For example, if a feature is a color (such as red, green, or blue), then one-hot encoding can make three columns named `is_red`, `is_green`, and `is_blue`. For each row, set one color to the value `1` and the other colors to `0`. This example uses the `lr_input` table, which represents the academic performance of students. ```sql SQL theme={null} CREATE TABLE mldemo.lr_input AS ( SELECT hours_studied, grade_in_prev_course, CASE WHEN RAND() > (1.0 - effective_score) THEN true ELSE false END AS got_top_marks FROM ( SELECT hours_studied, grade_in_pr ev_course, (2 * grade_in_prev_course + hours_studied) / 10.0 AS effective_score FROM ( SELECT mod(a.c1, 10) AS hours_studied, mod(b.c1, 5) AS grade_in_prev_course FROM sys.dummy100 a, sys.dummy100 b ) ) ); ``` There are two features: the number of hours spent studying `hours_studied` and the grade in the previous course `grade_in_prev_course`. Both input features are numeric. There are also two classes: `true` and `false`. Class values do not need to be Boolean, and they can be any distinct values. Students who have better grades in the previous course and spend more hours studying are more likely to have top marks in the course. However, this table also includes a randomness factor, which means that rows with the same features can have different classifications. This is normal in real-world data. This example creates the logistic regression model on the table, including the `metrics` option. This option causes the model to calculate the percentage of samples that are correctly classified. The Ocient System puts this information into the system catalog tables. ```sql SQL theme={null} CREATE MLMODEL lr1 TYPE LOGISTIC REGRESSION ON ( SELECT * FROM mldemo.lr_input ) options('metrics'->'true'); ``` This query selects the `machine_learning_models` and `logistic_regression_models` system catalog tables. ```sql SQL theme={null} SELECT name, on_select, num_arguments, zero_case, one_case, correctly_classified FROM sys.machine_learning_models a, sys.logistic_regression_models b WHERE a.id = b.machine_learning_model_id AND name = 'lr1'; name on_select num_arguments zero_case one_case correctly_classified ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ lr1 select * from mldemo.lr_input 2 cast('true' as BOOLEAN) cast('false' as BOOLEAN) 0.8397 Fetched 1 row ``` The model captures the two values for the two target classes. The `correctly_classified` column shows the model has almost 84 percent accuracy from working on the training data. This level of accuracy is pretty good, especially for a model using training data with significant randomness. This model can predict which classification is most likely if you provide the number of hours studied and grades in the previous course. First, review the average hours spent studying for each classification. ```sql SQL theme={null} SELECT avg(hours_studied) FROM mldemo.lr_input WHERE got_top_marks; (sum(hours_studied)) /(null_if(double(count(hours_studied)), (0))) ------------------------------------------------------------------ 5.231264244536801 Fetched 1 row SELECT avg(hours_studied) FROM mldemo.lr_input WHERE NOT got_top_marks; (sum(hours_studied)) /(null_if(double(count(hours_studied)), (0))) ------------------------------------------------------------------ 2.3534041715859897 Fetched 1 row ``` As expected, the rows where `got_top_marks` is true have a higher average value for `hours_studied`. The same should be true with `grade_in_prev_course`. ```sql SQL theme={null} SELECT AVG(grade_in_prev_course) FROM mldemo.lr_input WHERE got_top_marks; (SUM(grade_in_prev_course)) /(NULL_IF(DOUBLE(COUNT(grade_in_prev_course)), (0))) -------------------------------------------------------------------------------- 2.3535326451266925 Fetched 1 row SELECT avg(grade_in_prev_course) FROM mldemo.lr_input WHERE NOT got_top_marks; (sum(grade_in_prev_course)) /(NULL_IF(DOUBLE(COUNT(grade_in_prev_course)), (0))) -------------------------------------------------------------------------------- 0.9622195985832349 Fetched 1 row ``` You can also query to find the rows that are anomalies. In most cases, students achieved top marks in their class despite having minimal study hours and low grades in their previous classes. ```sql SQL theme={null} SELECT * FROM mldemo.lr_input WHERE got_top_marks != lr1(hours_studied, grade_in_prev_course) LIMIT 10; hours_studied grade_in_prev_course got_top_marks ------------------------------------------------------- 1 1 true 1 1 true 1 4 false 1 1 true 1 2 true 1 0 true 1 2 true 1 1 true 1 2 true 1 1 true Fetched 10 rows ``` For details about using this model type, see [Logistic Regression](/classification-models#logistic-regression). ## Support Vector Machine Support vector machine (SVM) tries to find a hyperplane to divide data into two classes with these objectives: * The hyperplane has the maximum margin, meaning the most distance between the two classes. * The model correctly classifies the highest percentage of data points. By default, SVM models in Ocient try to balance these two objectives. You can increase the priority of either objective by using the `regularizationCoefficient` option. For data sets with two features, the SVM hyperplane represents a straight line on a Cartesian graph. For three features, the hyperplane represents a plane. This example uses the SVM model on the same classification data set of student performance as used in the logistic regression section. ```sql SQL theme={null} CREATE MLMODEL svm1 TYPE SUPPORT VECTOR MACHINE ON ( SELECT * FROM mldemo.lr_input ) options('metrics'->'true'); Modified 0 rows ``` The `machine_learning_models` and `support_vector_machine_models` system catalog tables for SVM contain almost the same information as logistic regression. ```sql SQL theme={null} SELECT name, on_select, num_arguments, negative_case, positive_case, correctly_classified FROM sys.machine_learning_models a, sys.support_vector_machine_models b WHERE a.id = b.machine_learning_model_id AND name = 'svm1'; name on_select num_arguments negative_case positive_case correctly_classified ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ svm1 select * from mldemo.lr_input 2 cast('false' as BOOLEAN) cast('true' as BOOLEAN) 0.8397 Fetched 1 row ``` The system catalog tables indicate the same percentage of correctly classified rows as the logistic regression model (about 84 percent). The SVM model can increase the accuracy further by using a kernel. A kernel is a transformation to make data linearly separable. SVM in Ocient allows you to specify any kernel of your choice. For example, suppose that a circle can separate two classes of data, dividing data points between those inside and outside the circle. The SVM model cannot find a line to divide the two classes very well, or this problem is not linearly separable. However, you can transform the data into three dimensions (three features) to make it linearly separable. This is the idea of a kernel. You specify the kernel using the options `function1`, `function2`, `function3`, etc. Each function defines the formula for the kernel to create new features. Functions can use features from the original data set, represented by `x1`, `x2`, `x3`, etc. To explain this, revisit the circle example. Suppose that the circle centered at the point `(2, 2)` is the class inside. Otherwise, the circle is the class outside. The data that is separable by a circle in two dimensions is linearly separable in three dimensions with the features `x1`, `x2`, `x1^2 + x2^2`. This statement creates the data and the model. ```sql SQL theme={null} CREATE TABLE mldemo.circle AS ( SELECT x, y, CASE WHEN SQRT((x -2) ^ 2 + (y -2) ^ 2) <= 1 then 'INSIDE' ELSE 'OUTSIDE' END AS label FROM ( SELECT RAND() * 10 as x, RAND() * 10 as y FROM sys.dummy10000 ) ); Modified 10000 rows CREATE MLMODEL svm2 type SUPPORT VECTOR MACHINE ON ( SELECT x, y, label FROM mldemo.circle ) options( 'metrics'->'true', 'function1'->'x1', 'function2'->'x2', 'function3'->'x1^2 + x2^2' ); Modified 0 rows ``` The example includes three functions, meaning that the kernel increases from two to three dimensions. The kernel functions refer to the original feature values `x1` and `x2`. If you examine the `correctly_classified` value in the `machine_learning_models` and `support_vector_machine_models` system catalog tables, you can see a model that classifies nearly everything correctly. ```sql SQL theme={null} SELECT NAME, negative_case, positive_case, correctly_classified FROM sys.machine_learning_models a, sys.support_vector_machine_models b WHERE a.id = b.machine_learning_model_id AND name = 'svm2'; name negative_case positive_case correctly_classified ------------------------------------------------------------------------------------------------------------------------------------------------------------- svm2 cast('OUTSIDE' as CHAR) cast('INSIDE' as CHAR) 0.9913 Fetched 1 row ``` One advantage of SVM kernels is that they are automatic. The model saves all the kernel information and transforms the data as needed. Hence, you can execute this model to predict a new point without redefining the kernel. You can pass in the two features the model was built over, and a prediction comes out. The [sys.machine\_learning\_model\_options](/system-catalog#sys-machine_learning_model_options) table is the location where the SVM model saves kernel information. This table saves all the options that you specify. Some models also add additional keys to this dictionary to save certain information needed for when the model is executed. ```sql SQL theme={null} SELECT svm2(2, 2); COALESCE(('INSIDE'), (NULL)); --------------------------------------------- INSIDE Fetched 1 row ``` For details about using this model type, see [Support Vector Machine](/classification-models#support-vector-machine). ## Binary Classification with Neural Networks A neural network using the `FEEDFORWARD NETWORK` model type can perform binary classification. To use this model for classification, you must either convert labels to either `0` and `1` (using the `log_loss` option) or convert labels to `-1` and `1` (using the `hinge_loss` option). Both options can achieve the same result. This model can handle more complex cases, but using a different model for linearly separable data might be easier. In other words, a neural network model is useful when data is difficult to separate linearly using a kernel. This example shows a feedforward network model using the `log_loss` option. Note the `CASE` statement to convert labels to `0` or `1`. ```sql SQL theme={null} CREATE MLMODEL logloss TYPE FEEDFORWARD NETWORK ON ( SELECT x, y, CASE WHEN label = 'INSIDE' THEN 0 ELSE 1 END AS target FROM mldemo.circle ) options( 'metrics'->'true', 'hiddenLayers'->'2', 'hiddenLayerSize'->'2', 'outputs'->'1', 'lossFunction'->'log_loss' ); Modified 0 rows ``` This query finds that the classification accuracy is about 97 percent. ```sql SQL theme={null} SELECT COUNT(*) / 10000.0 FROM ( SELECT x, y, label, CASE WHEN logloss(x, y) = 0 then 'INSIDE' ELSE 'OUTSIDE' END AS predicted FROM mldemo.circle ) WHERE label = predicted; (_count(*)_0)/((10000.0)) -------------------------- 0.9698 Fetched 1 row ``` To use the `hinge_loss` option, use `-1` and `1` in the `CASE` statement. ```sql SQL theme={null} CREATE MLMODEL hingeloss TYPE FEEDFORWARD NETWORK ON ( SELECT x, y, CASE WHEN label = 'INSIDE' THEN -1 ELSE 1 END AS target FROM mldemo.circle ) options( 'metrics'->'true', 'hiddenLayers'->'2', 'hiddenLayerSize'->'2', 'outputs'->'1', 'lossFunction'->'hinge_loss' ); Modified 0 rows ``` The accuracy of the `hinge_loss` version is exactly the same as the `log_loss` version. ```sql SQL theme={null} SELECT count(*) / 10000.0 FROM ( SELECT x, y, label, CASE WHEN hingeloss(x, y) = -1 THEN 'INSIDE' ELSE 'OUTSIDE' END AS predicted FROM mldemo.circle ) WHERE label = predicted; (_count(*)_0)/((10000.0)) -------------------------- 0.9698 Fetched 1 row ``` For details about using this model type, see [Feedforward Neural Network](/other-models#feedforward-neural-network). ## K-Nearest Neighbors K-nearest neighbors (KNN) can only use numeric features, but KNN handles multi-classification in the Ocient System. This algorithm can handle an arbitrary number of target classes. One drawback with KNN is that it does not work at training time other than to take a snapshot of the training data. The algorithm operates on the data when you execute the model, so it can be a slow model for large data sets. To demonstrate KNN functionality, this example uses a similar circle of size (2, 2) with some modifications to use more classes: * Data within a unit circle of (2, 2) is classified as `INSIDE`. * Data within a radius of 2 is classified as `MIDDLE`. * Any other data is classified as `OUTSIDE`. ```sql SQL theme={null} CREATE TABLE mldemo.circles as ( SELECT x, y, CASE WHEN SQRT((x -2) ^ 2 + (y -2) ^ 2) <= 1 THEN 'INSIDE' WHEN SQRT((x -2) ^ 2 + (y -2) ^ 2) <= 2 THEN 'MIDDLE' ELSE 'OUTSIDE' END AS label FROM ( SELECT RAND() * 10 as x, RAND() * 10 as y FROM sys.dummy10000 ) ); Modified 10000 rows ``` KNN works by finding the `k` nearest points to the one the model is trying to predict a classification for. `k` is an integer value that you must specify during model creation. The model calculates distances using normal Euclidean distances. You can override the definition of distance by specifying a formula using the `distance` option. After the model identifies the points, it computes a score for each class and returns the class with the highest score. The `weight` option can influence each score for the specified point. By default, the `weight` value is based on the inverse of the distance (`1.0/(d+0.1)`). In other words, a point closer to the point you are trying to predict has a larger influence on the output result than a point further away. Any point outside of the nearest `k` points has no influence. This example uses three classes. ```sql SQL theme={null} CREATE MLMODEL knn1 TYPE K NEAREST NEIGHBORS ON ( SELECT x, y, label FROM mldemo.circles ) options('k'->'3'); Modified 0 rows ``` These queries test some values to see if the classifications are correct. ```sql SQL theme={null} SELECT knn1(2, 2); --------------------------------------------- INSIDE Fetched 1 row SELECT knn1(0.5, 2); --------------------------------------------- MIDDLE Fetched 1 row SELECT knn1(5,5); --------------------------------------------- OUTSIDE Fetched 1 row ``` KNN models perform work when you execute the model, and the amount of work increases as the input data increases. One solution to this is data reduction, which means finding a smaller subset of rows to build the KNN model over, enough to ensure that the accuracy of the model is still high. This method front-loads the work when you create the model so that it is much faster to execute on new data. To use the data reduction feature, set the `dataReduction` option to `true`. By default, the execution stops when the database finds a model that is 90 percent accurate over the training data or when the reduced model increases to 1,000 rows of data. You can override these defaults by setting the `maxReducedRows` or `targetAccuracy` options. Additionally, if you are using data reduction, you can ask the database to collect metrics on the percent of rows correctly classified by the model because this is much faster to do on the reduced data set. This example builds another KNN model over the same data and uses data reduction to find a model that is at least 95 percent accurate. ```sql SQL theme={null} CREATE MLMODEL knn2 TYPE K NEAREST NEIGHBORS ON ( SELECT x, y, label FROM mldemo.circles ) options( 'k'->'3', 'dataReduction'->'true', 'metrics'->'true', 'targetAccuracy'->'0.95' ); Modified 0 rows ``` Query the `machine_learning_models` and `k_nearest_neighbor_models` system catalog tables to see the snapshot table used for the model and its accuracy. ```sql SQL theme={null} SELECT table_name, correctly_classified FROM sys.machine_learning_models a, sys.k_nearest_neighbor_models b WHERE a.id = b.machine_learning_model_id AND name = 'knn2'; table_name correctly_classified ------------------------------------------------------------------- temp.knn86947930347148676 0.9541 Fetched 1 row ``` The model with data reduction correctly classifies about 95 percent of the training data. To see the size of the KNN model snapshot, you can query it by its table name. ```sql SQL theme={null} SELECT COUNT(*) FROM temp.knn86947930347148676; count(*) -------------------- 300 Fetched 1 row ``` The snapshot is only 300 rows. Without data reduction, the row count would be much larger by orders of magnitude. For details about using this model type, see [K Nearest Neighbor Classification](/classification-models#k-nearest-neighbors-classification). ## Multi-Class Classification with Neural Networks The `FEEDFORWARD NETWORK` model type can also perform multi-class classification when the feature classes are numeric. Multi-class classification works by using vectors, where each position in the vector indicates the score for that class. The [argmax](https://en.wikipedia.org/wiki/Arg_max) value of the score of each vector returns the class index. Much of this example is similar to the previous [Binary Classification with Neural Networks](#binary-classification-with-neural-networks) tutorial, with these modifications to the model options: * Set the `lossFunction` option to `cross_entropy`. * Set the `outputs` option to specify the number of classes for the model to use (in this example: `3`). * Set the `useSoftMax` option to `true`, which applies a [softmax](https://en.wikipedia.org/wiki/Softmax_function) function to the final vector. When the loss function is `cross_entropy_loss` , the option `useSoftMax` defaults to `true`. ```sql SQL theme={null} CREATE MLMODEL crossentropy TYPE FEEDFORWARD NETWORK ON ( SELECT x, y, CASE WHEN label = 'INSIDE' THEN { { 1.0, 0.0, 0.0 } } WHEN label = 'MIDDLE' THEN { { 0.0, 1.0, 0.0 } } ELSE { { 0.0, 0.0, 1.0 } } END AS TARGET FROM mldemo.circles ) options( 'metrics'->'true', 'hiddenLayers'->'2', 'hiddenLayerSize'->'2', 'outputs'->'3', 'lossFunction'->'cross_entropy_loss', 'useSoftMax'->'true' ); Modified 0 rows ``` Execute this model to return a vector. ```sql SQL theme={null} SELECT crossentropy(5,5) AS predicted; predicted -------------------------------------------------------------------------------- [[1.3436428911280516E-7, 1.811726873154417E-7, 0.9999996844630236]] Fetched 1 row ``` To get the classification, use the `VECTOR_ARGMAX` function to retrieve the index of the largest value. This function returns the 1‑based index of the largest value in the vector. ```sql SQL theme={null} SELECT VECTOR_ARGMAX(crossentropy(5,5)) AS predicted; predicted -------------------- 3 Fetched 1 row ``` This query finds the overall accuracy of the model. ```sql SQL theme={null} SELECT count(*) / 10000.0 FROM mldemo.circles WHERE ( label = 'INSIDE' AND VECTOR_ARGMAX(crossentropy(x, y)) = 1 ) OR ( label = 'MIDDLE' AND VECTOR_ARGMAX(crossentropy(x, y)) = 2 ) OR ( label = 'OUTSIDE' AND VECTOR_ARGMAX(crossentropy(x, y)) = 3 ); (_count(*)_0)/((10000.0)) -------------------------- 0.9298 Fetched 1 row ``` The model is about 93 percent effective. You can improve the accuracy by modifying the [Model Options](/other-models#model-options-2), including: * `hiddenLayers` * `hiddenLayerSize` * `lossFuncNumSamples` Increasing the resources of these options can improve accuracy, but could increase the time needed to train the model. For details about using this model type, see [Feedforward Neural Network](/other-models). ## Multi-Class Classification with Support Vector Machines and Logistic Regression Both logistic regression and SVM are binary classifiers, but you can use either to build a multi-class classifier to handle any number of classes. OcientML can perform this automatically with either model. This example shows the `CREATE` statement for the SVM model, which uses the same data as the [K Nearest Neighbors](#k-nearest-neighbors) and [Multi-Class Classification with Neural Networks](#multi-class-classification-with-neural-networks) examples. ```sql SQL theme={null} CREATE MLMODEL circ1 TYPE SUPPORT VECTOR MACHINE ON ( SELECT x, y, label FROM mldemo.circles ) options( 'metrics'->'true', 'function1'->'x1', 'function2'->'x2', 'function3'->'x1^2 + x2^2' ); Modified 0 rows ``` This statement uses the same kernel as the example for performing binary classification on [Support Vector Machines](#support-vector-machine). This query checks the metrics in the `machine_learning_models` and `support_vector_machine_models` system catalog tables to see the accuracy. ```sql SQL theme={null} SELECT correctly_classified FROM sys.machine_learning_models a, sys.support_vector_machine_models b WHERE a.id = b.machine_learning_model_id AND name = 'circ1'; correctly_classified ---------------------- 0.9579 Fetched 1 row ``` This model is slightly more accurate than the neural network classifier. This example tries logistic regression. ```sql SQL theme={null} CREATE mlmodel circ2 TYPE LOGISTIC REGRESSION ON ( SELECT x, y, label FROM mldemo.circles ) options('metrics'->'true'); Modified 0 rows SELECT correctly_classified FROM sys.machine_learning_models a, sys.logistic_regression_models b WHERE a.id = b.machine_learning_model_id AND NAME = 'circ2'; correctly_classified ---------------------- 0.9028 Fetched 1 row ``` Logistic regression shows lower accuracy than the neural network model. These classification models can handle non-numeric data. ## Naive Bayes The first of these models is naive Bayes. This model can handle binary or multi-class classification and features of any data type. By default, naive Bayes treats each feature as a discrete value. For example, if you assign features with values of `1`, `2`, `3`, and `4`, the model does not give any numeric significance to them. In this case, this model cannot predict the class for the sample with a feature value of `3.5` because that value is not explicitly assigned as a feature. However, the naive Bayes model can define certain input features as continuous features. These should be numeric values. In this case, the naive Bayes model finds the best-fit normal distribution for that feature to do the probability calculations. Naive Bayes models essentially use the probability rules of Bayes' Theorem to compute the most likely class based on the training data. The rules are only correct if the features are independent (uncorrelated). However, in practice, the models tend to work well with real-world data. This example uses this simple table to demonstrate naive Bayes models. ```sql SQL theme={null} SELECT * FROM mldemo.cars; color type origin stolen ---------------------------------------------------------------------------------------------------------------------------------------------- yellow sports domestic false yellow sports imported true yellow suv imported false red sports domestic false yellow suv domestic false red suv imported false red sports domestic true yellow suv imported true red sports imported true red sports domestic true Fetched 10 rows ``` `color`, `type`, and `origin` are the input features, which are all strings. `true` and `false` are the class labels. In this case, all of the input features are discrete. This example builds the naive Bayes model. ```sql SQL theme={null} CREATE MLMODEL nb1 TYPE NAIVE BAYES ON ( SELECT color, type, origin, stolen FROM mldemo.cars ) options('metrics'->'true'); Modified 0 rows ``` This query examines the information in the `machine_learning_models` and `naive_bayes_models` system catalog tables for naive Bayes models. ```sql SQL theme={null} SELECT NAME, on_select, num_arguments, result_probability_table, feature_result_matrix_table, correctly_classified FROM sys.machine_learning_models a, sys.naive_bayes_models b WHERE a.id = b.machine_le arning_model_id AND NAME = 'nb1'; name on_select num_arguments result_probability_table feature_result_matrix_table correctly_classified ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ nb1 select color, "type", origin, stolen from user.nb_input 3 temp.rpt83581840700868656 temp.frm83581860860782808 0.8 Fetched 1 row ``` In the model, the `correctly_classified` value indicates the accuracy is 80 percent, meaning just two rows are classified incorrectly. The two table names in the `temp` schema hold all the necessary data for executing the model as a function. This query calls the model to classify new data. ```sql SQL theme={null} SELECT nb1('red', 'suv', 'domestic') AS prediction; prediction ----------- false Fetched 1 row ``` Here is another example with a different data set to show how continuous features work with naive Bayes models. ```sql SQL theme={null} SELECT * FROM mldemo.golf; humidity play -------------------------- 70 false 75 true 86 true 95 false 80 true 85 false 96 true 90 true 91 false 90 false 70 true 70 true 80 true 65 true Fetched 14 rows ``` In this case, the only input feature is a continuous feature, but you can have multiple continuous features, and you can have a mix of continuous and discrete features. To build a naive Bayes model over this data, you must set which input features are continuous in the model. ```sql SQL theme={null} CREATE MLMODEL nb2 TYPE NAIVE BAYES ON ( SELECT * FROM mldemo.golf ) options('continuousFeatures'->'1', 'metrics'->'true'); Modified 0 rows ``` The `continuousFeatures` option is a comma-separated list (no spaces) of input column indexes that are continuous. Indexes start at 1. This query checks the `correctly_classified` value in the `machine_learning_models` and `naive_bayes_models` system catalog tables. ```sql SQL theme={null} SELECT correctly_classified FROM sys.machine_learning_models a, sys.naive_bayes_models b WHERE a.id = b.machine_learning_model_id AND name = 'nb2'; correctly_classified ---------------------- 0.7142857142857143 Fetched 1 row ``` Naive Bayes is a powerful classification algorithm, but it operates as a black box, making it hard to assess how it makes classification decisions. For details, see [Naive Bayes Classification](/classification-models#naive-bayes-classification). ## Decision Trees Unlike naive Bayes, the decision tree model allows you to see how the model works and makes its decisions. A decision tree branches based on the value of a single feature, while the leaves of the tree are the output classes. For SQL syntax, this model uses multiple nested `CASE` statements. You can examine the whole nested `CASE` statement that defines the model, but it might be very long. The decision tree model in Ocient has limited support for continuous features. The model finds the average value of the feature and allows a two-way branch based on whether the value is lower or higher than the average. Creating a decision tree model is exactly like a naive Bayes model, except with a different type. Decision trees also use the `continuousFeatures` option, which works the same way. These decision tree examples use the same data sets as the naive Bayes examples. ```sql SQL theme={null} CREATE MLMODEL dt1 TYPE DECISION TREE ON ( SELECT color, "type", origin, stolen FROM mldemo.cars ) options('metrics'->'true'); Modified 0 rows ``` This query examines the `machine_learning_models` and `decision_tree_models` system catalog tables. ```sql SQL theme={null} SELECT correctly_classified FROM sys.machine_learning_models a, sys.decision_tree_models b WHERE a.id = b.machine_learning_model_id AND name = 'dt1'; correctly_classified ---------------------- 0.8 Fetched 1 row ``` This model has the same 80 percent accuracy as the naive Bayes model. This query examines the `CASE` statement that defines the model. ```sql SQL theme={null} SELECT case_statement FROM sys.machine_learning_models a, sys.decision_tree_models b WHERE a.id = b.machine_learning_model_id AND name = 'dt1'; case_statement --------------------------------------------- CASE WHEN x3 = cast('domestic' as CHAR) THEN CASE WHEN x1 = cast('red' as CHAR) THEN CASE WHEN x2 = cast('sports' as CHAR) THEN cast('true' as BOOLEAN) ELSE cast('false' as BOOLEAN) END WHEN x1 = cast('yellow' as CHAR) THEN cast('false' as BOOLEAN) ELSE cast('false' as BOOLEAN) END WHEN x3 = cast('imported' as CHAR) THEN CASE WHEN x2 = cast('suv' as CHAR) THEN CASE WHEN x1 = cast('red' as CHAR) THEN cast('false' as BOOLEAN) WHEN x1 = cast('yellow' as CHAR) THEN cast('true' as BOOLEAN) ELSE cast('false' as BOOLEAN) END WHEN x2 = cast('sports' as CHAR) THEN cast('true' as BOOLEAN) ELSE cast('false' as BOOLEAN) END ELSE cast('false' as BOOLEAN) END Fetched 1 row ``` Like with the naive Bayes model, the decision tree model can predict new data. ```sql SQL theme={null} SELECT dt1('red', 'suv', 'domestic') AS prediction; prediction ----------- false Fetched 1 row ``` This example builds a model over the golfing data. ```sql SQL theme={null} CREATE mlmodel dt2 TYPE DECISION TREE ON ( SELECT * FROM mldemo.golf ) options('continuousFeatures'->'1', 'metrics'->'true'); Modified 0 rows ``` Review the `machine_learning_models` and `decision_tree_models` system catalog tables to see the accuracy of this model. ```sql SQL theme={null} SELECT correctly_classified FROM sys.machine_learning_models a, sys.decision_tree_models b WHERE a.id = b.machine_learning_model_id AND name = 'dt2'; correctly_classified ---------------------- 0.7142857142857143 Fetched 1 row ``` Again, the decision tree shows similar accuracy to the naive Bayes model. ## Related Links [Machine Learning Model Functions](/machine-learning-model-functions) [Classification Models](/classification-models) # Classification Models Source: https://docs.ocient.com/classification-models Train and use OcientML classification models such as decision trees, random forests, and naive Bayes in SQL to categorize data and detect patterns. supports classification models that involve understanding and grouping large data sets into preset categories or subpopulations. With the help of pre-classified training data sets, machine learning classification models leverage various algorithms to classify future data sets into respective and relevant categories. To create the model, use the `CREATE MLMODEL` syntax. For details, see [CREATE MLMODEL](/machine-learning-model-functions). Model option names are case-sensitive. ## K-Nearest Neighbors Classification Model Type: `K NEAREST NEIGHBORS` K-nearest neighbors (KNN) is a classification algorithm, where the first `N - 1` inputs are the features, which must be numeric. The last input column is a label, which can be any data type. There is no training step for KNN. Instead, when you create the model, the model saves a copy of all input data to a table, so that when the model is executed in a later SQL statement, a snapshot of the data the model is supposed to use is available. You can override both the weight function and the distance function. The system validates the `weight` and `distance` options during prediction time. If the values are invalid, the model throws an error during this time. ### **Model Options** #### Required `k` — This option must be a positive integer that specifies how many closest points to use for classifying a new point. #### Optional `distance` — If you specify this option, the value must be a function in SQL syntax for calculating the distance between a point used for classification and points in the training data set. This function should use the variables x1, x2, …​ for the 1st, 2nd, … features in the training data set, and p1, p2, …​ for the features in the point for classification. If you do not specify this option, the option defaults to the Euclidean distance. `metrics` — If you set this option to `true`, the model calculates the percentage of samples that are correctly classified by the model and saves this information in a catalog table. The default value is `false`. `normalize` — If you set this option to `true`, the model automatically computes the mean and standard deviation of each feature and uses them to normalize the data during training. The default value is `true`. `weight` — If you specify this option, the value must be a function in SQL syntax for calculating the weight of a neighbor. The function should use the variable d for distance. By default, the distance is set to 1.0/(d+0.1), thus avoiding division by zero on exact inputs and still allowing neighbors to have some influence. `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ### **Execute the Model** Create a KNN model with `8` closest points for classification and distance function `power(x1 - p1, 2) + power(x2 - p2, 2) + power(x3 - p3, 2)`. ```sql SQL theme={null} CREATE MLMODEL my_model TYPE K NEAREST NEIGHBORS ON ( SELECT x1, x2, x3, y1 FROM public.my_table ) options( 'k' -> '8', 'distance' -> 'power(x1 - p1, 2) + power(x2 - p2, 2) + power(x3 - p3, 2)' ); ``` After you create the model, you can see its details by querying the `sys.machine_learning_models` and `sys.machine_learning_model_options` system catalog tables. When you execute the model, the model executes with `N - 1` features as input and returns a label. The model chooses the label from the class with the highest score. The model scores classes by summing the weights from the nearest k points in the training data. ```sql SQL theme={null} SELECT my_model(x1, x2, x3) FROM my_table; ``` After you execute a model, you can find the details about the execution results in the `sys.k_nearest_neighbors_models` system catalog table. For details, see the description of the associated system catalog tables in the Machine Learning section in the [System Catalog](/system-catalog#page-title). ## Naive Bayes Classification Model Type: `NAIVE BAYES` Naive Bayes is a classification algorithm. The input is `N - 1` feature columns, and the last column is a label column. All columns can be any data type. The label column must be discrete. The feature columns can be discrete or continuous. When you use continuous feature columns, you must specify which columns are continuous (see options). Naive Bayes works by assuming that all features are equally important in the classification and that there is no correlation between features. With those assumptions, the algorithm computes all frequency information and saves it in three tables that you create using SQL `SELECT` statements. ### **Model Options** #### Optional `metrics` — If you set this option to `true`, the model calculates the percentage of samples correctly classified by the model and saves this information in a catalog table. This option defaults to `false`. `continuousFeatures` — If you specify this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start at 1. `normalize` — If you set this option to `true`, the model automatically computes the mean and standard deviation of each feature and uses them to normalize the data during training. Defaults to `true`. `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ### **Execute the Model** Create a Naive Bayes model with feature indexes `1,3`. ```sql SQL theme={null} CREATE MLMODEL my_model TYPE NAIVE BAYES ON ( SELECT x1, x2, x3, y1 FROM public.my_table ) options( 'continuousFeatures' -> '1,3' ); ``` After you create the model, you can see its details by querying the `sys.machine_learning_models` and `sys.machine_learning_model_options` system catalog tables. When you execute the model, you specify `N - 1` feature input arguments, and the model returns the most likely class. The returned class is based on computing the class with the highest probability, given prior knowledge of the feature values. In other words, the class `y` has the highest value of `P(y | x1, x2, …​, xn)`. ```sql SQL theme={null} SELECT my_model(col1, col2, col3) FROM my_table; ``` After you execute a model, you can find the details about the execution results in the `sys.naive_bayes_models` system catalog table. For details, see the description of the associated system catalog tables in the Machine Learning section in [System Catalog](/system-catalog). ## Decision Tree Model Type: `DECISION TREE` The decision tree is a classification model. The first `N - 1` input columns are features and can be any data type. All non-numeric features must be discrete and contain no more than the configured `distinctCountLimit` number of unique values. This limit is in place to prevent the internal model representation from growing too large. Numeric features are discrete by default and have the same limitation on the number of unique values, but they can be marked as continuous with the `continuousFeatures` option. For continuous features, the model builds the decision tree by dividing the values into two ranges instead of using discrete, unique values. The last input column is the label and can be any data type. When you create the model, you specify all features first, and then specify the label as the last column in the result set. You can use secondary indexes on discrete feature columns to greatly speed up the training of a decision tree model. supports a similar tree model for regression. For details, see [Regression Tree](/regression-models#regression-tree). ### **Model Options** #### Optional `continuousFeatures` — If you set this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start with 1. In the default state, the model considers no features as continuous. `distinctCountLimit` — If you set this option, the value must be a positive integer. This value sets the limit for how many distinct values a non-continuous feature and the label can contain. This option defaults to 256. `doPrune` — If you set this option to `true`, the model uses Pessimistic Error Pruning (PEP) to prune the tree after training. This option defaults to `false`. `featureArrayElements` — If you set this option, the value must be a comma-separated list of the feature indexes. If you set the `featureArray` option to `true`, this list determines the elements of the arrays to use for training. By default, the model uses all elements. `maxCellsToFetch` — If you set this option, the value must be a positive integer. Controls the chunking behavior when fetching feature values during model training. The limit represents the maximum number of data cells (calculated as number of columns × number of rows) that can be fetched in a single operation, not a byte limit. When the expected data size exceeds this threshold, the algorithm switches to database-based processing using SQL queries instead of in-memory processing. This value defaults to 33,554,432 cells (calculated as 32 \* 1024 \* 1024). `maxDepth` — If you set this option, the value must be a positive integer. This value sets the maximum allowable depth of the decision tree (the maximum number of features to split on). The default is unspecified, which means there is no maximum depth. `maxRows` — If you set this option, the value must be a positive integer. This option limits the number of rows used for model training by creating a snapshot table with only the specified number of rows from the input query. This option cannot be used with `noSnapshot -> true` (attempting to set both results in an `invalid argument` error during model creation). When this option is unspecified, the model trains using all rows from the input query. `maxThreads` — If you set this option, the value must be a positive integer. This value indicates the maximum number of parallel threads to use while the model trains. The default value is 2. `metrics` — If you set this option to `true`, the model also calculates the percentage of samples correctly classified by the model and saves this information in a catalog table. This option defaults to `false`. `noSnapshot` — If you set this option to `true`, the database does not create an intermediate table that stores the result of the specified SQL statement, which the model uses for training. This option defaults to `false`. In this case, the database creates and uses the intermediate table. Setting this option to `true` is useful when the training set is fixed. If the training set is a table with modifications, set this option to `false` as the decision tree trainer uses different data sets in different parts of the tree. Likewise, if the training set consists of a query that returns 100 rows, then set this option to `false` because there is no guarantee that running that query twice generates the same 100 rows each time. `numSplits` — If you set this option, the value must be an integer greater than 1. This value sets the maximum number of binary branches a continuous feature can consider. The default value is 32. `ROCNumSamples` — If you set the option, you must also set the `metrics` option. This positive integer indicates the number of samples for the model to use for the area under the ROC curve. The default value is 10. `skipLimitCheck` — If you set this option to `true`, the model skips cardinality checks that throw errors when columns have too many values. The limit that this option checks is the same one that is specified by the `distinctCountLimit` option. This option defaults to `false`. `weighted` — If you set this option, the model considers weights for labels. If you set this option value to `true`, you must specify an additional column as a `double` in the training data for label weights. Rows with the same labels must have the same weights. If you set this value to `auto`, the model calculates weights automatically by weighting each label according to the ratio of the count of the most frequent label to the count of the specified label. As a result, the most frequent label has a weight of `1.0`, and the other label weights are higher. This option defaults to `false`, which means all labels have equal weight. `splitMetric` — Controls which function is used to evaluate the quality of a split during tree construction. Supported options are: * `gini_impurity` (default) — Measures impurity based on class distributions. * `weighted_f1` — Uses class-frequency–weighted F1 score to guide splits. `enableResplits` — A Boolean value that determines if the tree can reuse the same continuous feature multiple times along a single branch (e.g., split on `x1 < 7` and later `x1 < 3`). This can capture more complex, range-specific relationships. If unspecified, this value defaults to true, meaning continuous features remain available for additional splits after use, allowing the tree to create more complex decision boundaries. When set to false, the model marks continuous features as exhausted after their first use, and they cannot be used again in subsequent splits in the same tree. `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ### **Execute the Model** Create a decision tree model with feature indexes `1,3`. ```sql SQL theme={null} CREATE MLMODEL my_model TYPE DECISION TREE ON ( SELECT c1, c2, c3, y1 FROM public.my_table ) options( 'continuousFeatures' -> '1,3' ); ``` After you create the model, you can see its details by querying the `sys.machine_learning_models` and `sys.machine_learning_model_options` system catalog tables. When you execute the model, you must specify the `N - 1` features as parameters. The model returns the expected label. ```sql SQL theme={null} SELECT my_model(col1, col2, col3) FROM my_table; ``` After you execute a model, you can find the details about the execution results in the `sys.decision_tree_models` system catalog table. For details, see the description of the associated system catalog tables in the Machine Learning section in [System Catalog](/system-catalog). ## Random Forest Model Type: `RANDOM FOREST` Random forest is a classification model consisting of multiple decision trees. The result consists of the most common label among the tree results and an array of pairs of labels and their frequencies, sorted in descending order by frequency. The model breaks ties in various ways depending on the type of label. For strings, the model uses lexicographic order (reverse of the usual alphabetic order), so `C` comes before `A`, for example. For Booleans, the model chooses `true` before `false`. For numeric types, the model chooses the largest number first. When you call the model to make classification predictions, you can optionally use soft voting by adding an extra Boolean argument to the statement, e.g., `house(x, y, true)`. This extra argument must be a Boolean literal, either `true` (soft voting) or `false` (hard voting). If you do not specify this value, the default value is `false` (hard voting). In soft voting, each decision tree reports a list of possible results and their confidence factors. The random forest model adds the confidence factors from the decision trees and normalizes them to add up to 1.0. The model sorts the results by descending confidence, then by descending values. In hard voting, the model does not account for confidence factors. Each decision tree makes its own class prediction as a vote. The model selects the prediction with the most votes from the trees. ### **Model Options** #### Required `numChildren` — Number of child decision trees. #### Optional `bootstrap` — If you set this option to `true`, the model uses bootstrap sampling with replacement, meaning each tree in the random forest is trained on a random subset of the data (either the `rowsPerChild` or `fractionSelected` value sets the exact number of rows), and the same row can appear multiple times in each tree. If you set this option to `false`, this option does not use replacement, meaning each row can appear at most once per tree. The default value is `false`. `continuousFeatures` — If you set this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start with 1. `distinctCountLimit` — If you set this option, the value must be a positive integer. This value limits how many distinct values a non-continuous feature and the label can contain. The default value is 256. `featureSubsetStrategy` — If you set this option, the model passes this option directly to the child decision trees to specify how many features each tree should consider at each split from the still-available features. When this value is higher, the model has higher accuracy and lower variance, but it takes longer to train. You can specify this option either as an integer (e.g., `4`, meaning consider up to four features at each split) or one of three string values: * `all` — Each tree checks every feature. * `sqrt` — Each tree checks up to the square root of the total number of features. * `one-third` — Each tree checks up to one-third of the total number of features. The default value is `all`. `fractionSelected` — The proportion of rows the model uses to train each decision tree. The value is a `double` that must be in the interval (0, 1]. You cannot set this option if you also set the `rowsPerChild` option to a positive value. The default behavior is that the model uses all available rows. `inputsPerChild` — Number of features used to create each child decision tree. The default value is the number of features you specify for the forest divided by 3 and rounded up. `maxCellsToFetch` — If you set this option, the value must be a positive integer. Controls the chunking behavior when fetching feature values during model training. The limit represents the maximum number of data cells (calculated as number of columns × number of rows) that can be fetched in a single operation, not a byte limit. When the expected data size exceeds this threshold, the algorithm switches to database-based processing using SQL queries instead of in-memory processing. This value defaults to 33,554,432 cells (calculated as 32 \* 1024 \* 1024). `maxChildThreads` — An integer representing the maximum number of threads each child decision tree can use. If you do not specify this option, each child decision tree uses at most one thread, the default for decision trees. `maxDepth` — If you set this option, the value must be a positive integer. This value sets the maximum allowable depth of the decision tree. The default value is 3. `maxThreads` — The maximum number of parallel threads to use while the model trains decision trees. This value must be a positive integer. The default value is 16. `metrics` — If you set this option to `true`, the model also calculates the percentage of samples that are correctly classified by the model for the random forest and saves this information in a catalog table. This option is always set to `false` for the child trees. `noSnapshot` — If you set this option to `true`, the data source must not change. In this case, the database does not create an intermediate table that stores the result of the specified SQL statement, which the model uses for training a random forest. Child decision trees always have this option set to `true`, so the database does not create a separate intermediate table for each decision tree. The default value is `false`. Setting this option to `true` is useful when the training set is fixed. If the training set is a table with modifications, set this option to `false` as the decision tree trainer uses different data sets in different parts of the tree. Likewise, if the training set consists of a query that returns 100 rows, then set this option to `false` because there is no guarantee that running that query twice generates the same 100 rows each time. `requiredFeatures` — A comma-separated list of integers as strings representing specific features where the first feature has the value `1`. The model uses these features in every decision tree in the forest. The default behavior is that the decision tree in the forest can train on any feature that is in the list. `rowsPerChild` — If you set this option to a positive integer, the number represents the number of rows (from a random sample) to use for each decision tree. If you set this option to 0, each child uses all available rows. The default value is 0. You cannot set this option to a positive value if you also set the `fractionSelected` option. `skipLimitCheck` — If you set this option to `true`, the model skips cardinality checks that throw errors when columns have too many values. The limit that this option checks is the same one that is specified by the `distinctCountLimit` option. The default value is `false`. `weighted` — If you set this option, the model considers weights for labels. If you set this option value to `true`, you must specify an additional column as a `double` in the training data for label weights. Rows with the same labels must have the same weights. If you set this value to `auto`, the model calculates weights automatically by weighting each label according to the ratio of the count of the most frequent label to the count of the specified label. As a result, the most frequent label has a weight of `1.0`, and the other label weights are higher. This option defaults to `false`, which means all labels have equal weight. `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ### **Execute the Model** Create a random forest model with four child decision trees and two features randomly chosen by the model for each tree. The `training_table` table contains the training data set. Collect metrics for the model execution by setting the `metrics` option to `true`. ```sql SQL theme={null} CREATE MLMODEL test_model TYPE RANDOM FOREST ON ( SELECT * FROM public.training_table ) options( 'numChildren' -> '4', 'inputsPerChild' -> '2' ); ``` After you create the model, you can see its details by querying the `sys.machine_learning_models`, `sys.machine_learning_model_options`, and `sys.random_forest_models` system catalog tables. Execute this model using three columns `a`, `b`, and `c`, from `large_table` table that contains the whole data set. ```sql SQL theme={null} SELECT test_model(a, b, c) FROM large_table; ``` Optionally, you can execute the model using soft voting by including an extra Boolean argument. ```sql SQL theme={null} SELECT test_model(a, b, c, true) FROM large_table; ``` The output of the model is a tuple containing the most common label as the first element. The second element is an array of tuples, where the first element is the label and the second is the frequency. The model sorts the array by frequency from highest to lowest, where the sum of frequencies is 1. When two labels have the same frequency, the model breaks the tie according to lexicographic order. To retrieve the most common label, use `[]` with an index of 1, `[1]`, for example, `test_model(a,b,c)[1]`. After you execute a model, you can find the results in the output of the model function execution. For details, see the description of the associated system catalog tables in the Machine Learning section in [System Catalog](/system-catalog). ## Logistic Regression Model Type: `LOGISTIC REGRESSION` This model fits a logistic curve to the data across any number of classes greater than one. The first `N - 1` inputs are features and must be numeric. Features can be one-hot encoded. The last input column is the class or label. You must have at least one non-NULL label in the result set for model creation. The model best fits the logistic curve using a negative log likelihood loss function. The model uses an algorithm that is a combination of particle swarm optimization, line search, and genetic algorithms to find the best-fit parameters. ### **Model Options** #### Optional `metrics` — If you set this option to `true`, the model calculates the percentage of samples that are correctly classified by the model and saves this information in the `sys.logistic_regression_models` system catalog table. This option defaults to `false`. `normalize` — If you set this option to `true`, the model uses auto-scaling to compute the mean and standard deviation of each input feature to normalize data during training, making training more numerically stable. The model then unscales parameters so the persisted model operates in the original units. This option defaults to `true`. `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `numEpochs` — An `INTEGER` value representing the maximum number of epochs, or full passes during training through the entire data set. This value must be positive. If you do not specify this value, the default is 20. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ### **Execute the Model** Create a logistic regression model. Collect metrics for the model execution by setting the `metrics` option to `true`. ```sql SQL theme={null} CREATE MLMODEL my_model TYPE LOGISTIC REGRESSION ON ( SELECT x1, x2, x3, y1 FROM public.my_table ) options( 'metrics' -> 'true' ); ``` After you create the model, you can see its details by querying the `sys.machine_learning_models` and `sys.machine_learning_model_options` system catalog tables. When you execute this model after training, you must specify the features as input and the label as the output. The label can be any data type. ```sql SQL theme={null} SELECT my_model(col1, col2, col3) FROM my_table; ``` After you execute a model, you can find the details about the execution results in the `sys.logistic_regression_models` system catalog table. For details, see the description of the associated system catalog tables in the Machine Learning section in the [System Catalog](/system-catalog). ## Support Vector Machine Model Type: `SUPPORT VECTOR MACHINE` Support Vector Machine (SVM) essentially finds a hypersurface (the hypersurface is a curve in 2-dimensional space) that correctly splits the data into any number of classes greater than one and maximizes the margin around the hypersurface. By default, SVM finds a hyperplane to split the data (the hyperplane is a straight line in 2-dimensional space). SVM uses a hinge loss function to balance the two objectives of finding a hyperplane with a wide margin while minimizing the number of incorrectly classified points. The first `N - 1` input columns are the features and must be numeric. The last column is the label and can be any arbitrary type. You must have at least one non-NULL label in the result set for model creation. ### **Model Options** #### Optional `metrics` — If you set this option to true, the model also calculates the percentage of samples that are correctly classified by the model and saves this information in a catalog table. This option defaults to false. `regularizationCoefficient` — If you specify this option, the value must be a valid floating-point number. This option is used to control the balance of finding a wide margin and minimizing incorrectly classified points in the loss function. When this value is larger (and positive), it makes having a wide margin around the hypersurface more important relative to the incorrectly classified points. Because of how Ocient implements SVM, the values for this parameter are likely different from the values used in other common SVM implementations. This option defaults to 1.0 / 1000000.0. `functionN` — By default, SVM uses a linear kernel. If you use a different kernel, you must provide a list of functions that are summed together, just like with linear combination regression. You must specify the first function using a key named 'function1'. Subsequent functions must use keys with names that use subsequent values of N. You must specify functions in SQL syntax and use the variables x1, x2, …​, xn to refer to the 1st, 2nd, and nth independent variables, respectively. You can specify the default linear kernel as: 'function1' → 'x1', 'function2' → 'x2', and so on. The model always adds a constant term equivalent to 'functionN' → '1.0' that you do not need to specify explicitly. `normalize` — If you set this option to `true`, the model automatically computes the mean and standard deviation of each feature and uses them to normalize the data during training. Defaults to `true`. `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `numEpochs` — An `INTEGER` value representing the maximum number of epochs, or full passes during training through the entire data set. This value must be positive. If you do not specify this value, the default is 200. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ### **Execute the Model** Create a support vector machine model. ```sql SQL theme={null} CREATE MLMODEL my_model TYPE SUPPORT VECTOR MACHINE ON ( SELECT c1, c2, c3, y1 FROM public.my_table ); ``` After you create the model, you can see its details by querying the `sys.machine_learning_models` and `sys.machine_learning_model_options` system catalog tables. When you execute the model, the `N - 1` features must be passed as parameters. The model returns the expected label. ```sql SQL theme={null} SELECT my_model(col1, col2, col3) FROM my_table; ``` After you execute a model, you can find the details about the execution results in the `sys.support_vector_machine_models` system catalog table. For details, see the description of the associated system catalog tables in the Machine Learning section in the [System Catalog](/system-catalog). ## Gradient Boosted Trees Model Type: `GRADIENT BOOSTED TREES` Gradient Boosted Trees (GBT) is an ensemble machine learning algorithm that builds a sequence of decision trees where each new tree is trained to correct the prediction errors of the previous trees. Unlike the Random Forest model, which creates trees independently in parallel, GBT uses a sequential approach based on gradient descent optimization, where each tree learns to predict the residual errors (gradients) of the current ensemble. This iterative error-correction process allows the model to capture complex, non-linear patterns by progressively refining predictions through multiple weak learners. The algorithm supports both regression and classification tasks through different loss functions (use the `lossFunction` option to toggle whether the model performs regression or classification tasks). For regression with squared error loss, trees directly predict residual errors, while classification with logistic loss maintains raw scores transformed through sigmoid or softmax functions. ### Model Options #### Required `numChildren` — An INTEGER value representing the total number of trees to build sequentially. Each tree learns to correct the errors of the previous trees. `learningRate` — A DECIMAL value between 0.0 and 1.0 that tunes how much the model learns from each successive child. #### Optional `lossFunction` — A string value that determines how the model calculates prediction errors and what type of problem it solves. Accepted values include: * `'squared_error'` — Configures the model for regression tasks. Calculates errors as the squared difference between predicted and actual values. Use this for predicting continuous numeric values (e.g., prices, temperatures, quantities). The target column must contain numeric values. This is the default value. * `'log_loss'` — Configures the model for classification tasks. This model uses logistic loss to calculate prediction errors for probability-based predictions. `fractionSelected` — A DECIMAL value greater than 0.0 and less than or equal to 1.0 that specifies what fraction of the training rows to randomly select for each boosting iteration. When you specify a value, the algorithm uses `CEIL(fractionSelected * totalRows)` per iteration. If you do not specify this value, the model uses all rows in their original order. `inputsPerChild` — An INTEGER greater than or equal to 1 that specifies the number of input features each boosting tree should use. This value cannot exceed the number of input features available in the data set. When you specify this value, the algorithm deterministically cycles through pre-enumerated feature subsets to ensure each tree uses exactly this many features. When you do not specify this value, the model uses all available features for each tree. `maxDepth` — A positive INTEGER value that represents the maximum allowable depth of the decision trees. The default value is 3. `maxThreads` — A positive INTEGER value that sets the maximum number of parallel threads to use for training each child decision tree. Parallel threads do not affect the sequential method of training each tree. The default value is 16. `enableResplits` — A Boolean value that determines if the tree can reuse the same continuous feature multiple times along a single branch (e.g., split on `x1 < 7` and later `x1 < 3`). This action can capture more complex, range-specific relationships. If you do not specify this value, the default value is `true`, meaning continuous features remain available for additional splits after use, which allows the tree to create more complex decision boundaries. When you set this value to `false`, the model marks continuous features as exhausted after their first use, and the model cannot use these features again in subsequent splits in the same tree. `resplitDepth` — An INTEGER value that sets the maximum depth at which tree nodes can be re-split during optimization. Controls how deep the algorithm searches for better split points. If you do not specify this value, the default value is 6. `resplitThreshold` — A DECIMAL value that sets the minimum improvement threshold required to trigger a re-split operation. Lower values (e.g., 0.01) allow more aggressive re-splitting but can increase training time. Higher values (e.g., 1.0) require larger improvements to trigger re-splits. If you do not specify this value, the default value is 0.1. `maxCellsToFetch` — An INTEGER value that determines the memory threshold to switch from training with system memory to training with SQL queries in the database. In-memory training is generally faster, but is limited by the available SQL Node memory. If the size of a training data subset exceeds this value, then the system performs training operations using SQL queries. The default value is 33,554,432 (calculated as `32 * 1024 * 1024`). `metrics` — A Boolean value. If you set this value to `true`, the system calculates and stores final model metrics (`R²/RMSE` for regression or `Accuracy/LogLoss` for classification) on the training data. `featureArray` — A Boolean value. If you set this value to `true`, the model expects all input features to be in a single ARRAY column instead of separate columns. `continuousFeatures` — A set of INTEGER values that specify the input feature columns that the model should treat as continuous numeric variables for optimal tree splitting. The value must be a comma-separated list of feature column indexes (starting with 1) that correspond to numeric columns in your `SELECT` SQL statement, excluding the target column. For example, `'continuousFeatures' -> '1,2'` would treat the first and second columns of your `SELECT` statement as continuous. If you do not specify this value, the model treats all features as categorical, which can impair results for numeric columns like prices, measurements, or scores. ### Execute the Model This example creates a Gradient Boosted Trees classifier to predict whether a customer stops using a service. The model trains on four input features (`tenure_months`, `monthly_charges`, `total_charges`, `support_tickets`) to predict the target variable (`churn_label`), filtering out any records with missing labels to ensure clean training. The model uses these options: * `'numChildren' -> '100'` — Builds a sequence of 100 weak learners (regression trees). Each tree learns to correct the classification residuals (logistic loss) of all previous trees. A higher value can capture complex patterns, and you should pair it with an appropriate learning rate. * `'learningRate' -> '0.1'` — Scales the contribution of each new tree by 10 percent, balancing convergence speed and generalization. This value represents a small learning rate, making training more conservative and stable. * `'lossFunction' -> 'log_loss'` — Configures the model for classification through logistic loss. If you do not specify this value, the default value is `'squared_error'` (regression), so explicitly setting `'log_loss'` is mandatory for classifiers. ```sql SQL theme={null} CREATE MLMODEL churn_gbt_classifier TYPE GRADIENT BOOSTED TREES ON ( SELECT tenure_months, monthly_charges, total_charges, support_tickets, churn_label FROM customer_training WHERE churn_label IS NOT NULL ) OPTIONS ( 'numChildren' -> '100', 'learningRate' -> '0.1', 'lossFunction' -> 'log_loss' ); ``` Call the trained model in a `SELECT` query, passing in the feature columns as arguments. ```sql SQL theme={null} SELECT customer_id, churn_gbt_classifier( tenure_months, monthly_charges, total_charges, support_tickets )[1] AS predicted_label, churn_gbt_classifier( tenure_months, monthly_charges, total_charges, support_tickets )[2] AS class_probabilities FROM customer_scoring WHERE tenure_months IS NOT NULL AND monthly_charges IS NOT NULL AND total_charges IS NOT NULL AND support_tickets IS NOT NULL; ``` For classification, the model returns a tuple of: * `element [1]` — The predicted class label * `element [2]` — An array of (`class_label`, `probability`) pairs For the output of the model, see the `sys.gradient_boosted_trees_models` system catalog table. ## Related Links [Clustering and Dimension Reduction Models](/clustering-and-dimension-reduction-models) [Other Models](/other-models) [Machine Learning Models](/machine-learning-models) # # Cluster and Node Management Source: https://docs.ocient.com/cluster-and-node-management Manage clusters and nodes in an Ocient System, including adding, removing, restarting, monitoring, and replacing storage, foundation, and SQL nodes. Cluster and Node Management commands allow administrators to manage storage clusters, create and manage storage spaces, create and manage nodes, and apply configuration changes to nodes, clusters, or the overall system. ## CLUSTER ### CREATE CLUSTER `CREATE CLUSTER` creates a new cluster in the current database. The cluster name must be distinct from the name of any existing tables in the database. To create a cluster, the logged-in user must be a system-level user. For details about creating clusters, see [Ocient Application Configuration](/ocient-application-configuration). For Foundation clusters, if the system is configured with a single-cluster configuration, when you add a second Foundation cluster, the system automatically changes to a multi-cluster configuration. This change is irreversible and requires a full system restart to fully take effect. **Syntax** ```sql SQL theme={null} CREATE CLUSTER [ IF NOT EXISTS ] cluster_name ::= TYPE [=] foundation | PARTICIPANTS [=] (node_name1, node_name2, ...) | STORAGESPACE [=] storage_space_name ``` | **Parameter** | **Data** **Type** | **Description** | | -------------- | ----------------- | -------------------------------------- | | `cluster_name` | string | The name of the new cluster to create. | #### Define Clusters ( `` ) You must include these configuration settings when you create a new cluster. | **Configuration** | **Data Type** | **Description** | | ----------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `TYPE` | string | The supported value is:
`foundation` — Processes query operators and handles storage on the system. | | `PARTICIPANTS` | string | A list of one or more node names included in the new cluster.
The number of participant nodes in a cluster must be equal to or greater than the storage-space width. For details, see [CREATE STORAGESPACE](#create-storagespace).
Adding more than the minimum number of participants to a cluster overprovisions the cluster and allows replacing or dropping participant nodes.
Nodes cannot be assigned as participants to multiple clusters. | | `STORAGESPACE` | string | The name of the storage space for the cluster. | **Examples** This example creates a Foundation cluster named `lts_cluster0` using the storage space named `my_storage_space` with three participant nodes. ```sql SQL theme={null} CREATE CLUSTER lts_cluster0 TYPE=foundation PARTICIPANTS=(node1,node2,node3) STORAGESPACE=my_storage_space; ``` ### DROP CLUSTER `DROP CLUSTER` removes an existing cluster from the system. To remove a cluster, you must be a system-level user. **Syntax** ```sql SQL theme={null} DROP CLUSTER [ IF EXISTS ] cluster_name ``` | **Parameter** | **Data** **Type** | **Description** | | -------------- | ----------------- | -------------------------------------------- | | `cluster_name` | string | The name of the specified cluster to remove. | **Example** This example removes an existing cluster named `lts_cluster0`. ```sql SQL theme={null} DROP CLUSTER lts_cluster0; ``` ### ALTER CLUSTER #### ALTER CLUSTER ADD PARTICIPANTS `ALTER CLUSTER ADD PARTICIPANTS` adds the specified nodes to an existing cluster in the system. This operation can overprovision the storage space if necessary. For the system to recognize newly added nodes, you must restart the cluster. To do this, you can follow the steps in [Add Foundation Nodes](/expand-and-rebalance-system#add-foundation-nodes). **Syntax** ```sql SQL theme={null} ALTER CLUSTER [ IF EXISTS ] cluster_name ADD PARTICIPANTS [=] ( node_name1 [,...] ); ``` | **Parameter** | **Data** **Type** | **Description** | | -------------------- | ----------------- | ---------------------------------------------------- | | `cluster_name` | string | The name of the specified cluster to alter. | | `node_name1, [,...]` | string | The name of one or more nodes to add to the cluster. | **Example** This example adds the node `node3` to a cluster named `example_cluster`, which already contains `node1` and `node2`. Following this operation, `example_cluster` consists of three nodes. ```sql SQL theme={null} ALTER CLUSTER example_cluster ADD PARTICIPANTS (node3); ``` #### ALTER CLUSTER DROP PARTICIPANTS `ALTER CLUSTER DROP PARTICIPANTS` drops nodes from an existing cluster. This SQL statement fails for a Foundation cluster if dropping the specified nodes brings the number of participants below the total width of the storage space associated with that cluster. **Syntax** ```sql SQL theme={null} ALTER CLUSTER [ IF EXISTS ] cluster_name DROP PARTICIPANTS [=] ( node_name1 [,...] ); ``` | **Parameter** | **Data** **Type** | **Description** | | -------------------- | ----------------- | ------------------------------------------------------- | | `cluster_name` | string | The name of the specified cluster to alter. | | `node_name1, [,...]` | string | The name of one or more nodes to drop from the cluster. | **Example** This example drops participant node `node3` from a cluster named `example_cluster`, which contains `node1`, `node2`, and `node3`. Following this operation, the participants in `example_cluster` consist of the nodes `node1` and `node2`. ```sql SQL theme={null} ALTER CLUSTER example_cluster DROP PARTICIPANTS (node3); ``` #### ALTER CLUSTER ADD STORAGESPACE `ALTER CLUSTER ADD STORAGESPACE` links an existing storage space to a storage cluster, enabling tables in that storage space to use the Foundation Nodes of that cluster. A storage space must be associated with at least one storage cluster before you can create tables in the space. **Required Privileges** To add a storage space to a cluster, you must have system-level privileges. **Syntax** ```sql SQL theme={null} ALTER CLUSTER [ IF EXISTS ] cluster_name ADD STORAGESPACE storage_space_name ``` | Parameter | Type | Description | | -------------------- | ------ | -------------------------------------------------------------------- | | `cluster_name` | string | The name of the storage cluster to modify. | | `storage_space_name` | string | The name of an existing storage space to associate with the cluster. | **Example** This example adds the storage space `analytics_space` to the cluster `lts_cluster_0`. ```sql SQL theme={null} ALTER CLUSTER lts_cluster_0 ADD STORAGESPACE analytics_space; ``` #### ALTER CLUSTER REMOVE STORAGESPACE `ALTER CLUSTER REMOVE STORAGESPACE` detaches a storage space from a storage cluster. This detaching prevents the creation of new tables in that storage space within the cluster. The statement fails if: * Tables still exist in the storage space within the cluster. Remove all associated tables before detaching the storage space. * The storage space is currently set as the system default storage space. Reset the default by using [ALTER SYSTEM SET DEFAULT STORAGESPACE](#alter-system-set-default-storagespace) before detaching the storage space. * You are attempting to detach the core system storage space: `"systemStorageSpace"`. **Required Privileges** To detach a storage space from a cluster, you must have system-level privileges. **Syntax** ```sql SQL theme={null} ALTER CLUSTER [ IF EXISTS ] cluster_name REMOVE STORAGESPACE storage_space_name ``` | Parameter | Type | Description | | -------------------- | ------ | --------------------------------------------------------- | | `cluster_name` | string | The name of the storage cluster to modify. | | `storage_space_name` | string | The name of the storage space to detach from the cluster. | **Example** This example detaches the storage space `analytics_space` from the cluster `lts_cluster_0`. ```sql SQL theme={null} ALTER CLUSTER lts_cluster_0 REMOVE STORAGESPACE analytics_space; ``` #### ALTER CLUSTER ALTER CONFIG SET `ALTER CLUSTER ALTER CONFIG SET` sets a configuration override for the configuration at the cluster scope. For an explanation of how configuration overrides work, see [Inspect the Current Configuration](/inspect-the-current-configuration). If the specified path to the configuration setting does not exist, the system considers it as a new configuration and appends it to the collection of existing configurations. To reset a configuration override for a specific key, use the `RESET` keyword. **Syntax** ```sql SQL theme={null} -- To set a new configuration override: ALTER CLUSTER [ IF EXISTS ] cluster_name ALTER CONFIG SET string_definition [=] string_value [,...] -- To reset a configuration override: ALTER CLUSTER [ IF EXISTS ] clusterName ALTER CONFIG RESET [ string_definition [,...] ] ``` | **Parameter** | **Data** **Type** | **Description** | | ------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `cluster_name` | string | The name of the specified cluster to alter. | | `string_definition` | string | The name of a configuration parameter to alter.

Contact support for details about altering configuration parameters. | | `string_value` | string | The new value to set for the specified configuration parameter. | **Examples** This example alters the base configuration for cluster `cluster_name` and sets the certificate for all nodes to a certificate named `cert_name.crt`. ```sql SQL theme={null} ALTER CLUSTER cluster_name ALTER CONFIG SET 'certificateStore.cert' = 'cert_name.crt'; ``` This example resets the certificate configuration for cluster `cluster_name`. ```sql SQL theme={null} ALTER CLUSTER cluster_name ALTER CONFIG RESET 'certificateStore.cert'; ``` This example alters the configuration at the cluster scope for cluster `cluster_name` and service role `lts`. ```sql SQL theme={null} ALTER CLUSTER cluster_name ALTER CONFIG SET 'lts.numLevels' = '3'; ``` #### ALTER CLUSTER ALTER LOG LEVEL SET `ALTER CLUSTER cluster_name ALTER LOG LEVEL SET` sets the log level for a particular logger for all nodes in the provided cluster. For an explanation of how configuration overrides work, see [Inspect the Current Configuration](/inspect-the-current-configuration). Syntax ```sql SQL theme={null} -- To set a new configuration override: ALTER CLUSTER cluster_name ALTER LOG LEVEL SET [ [,...] ] ::= { 'ALL' | logger_name [=] logging_level } -- To reset a configuration override: ALTER CLUSTER cluster_name ALTER LOG LEVEL RESET [logger_name [, ...] ] ``` | **Parameter** | **Data** **Type** | **Description** | | -------------- | ----------------- | ------------------------------------------- | | `cluster_name` | string | The name of the specified cluster to alter. | **Logger Parameters (**``**)** | **Configuration** | **Data Type** | **Description** | | ----------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `logger_name` | string | Use `ALL` for the root logging configuration.
Otherwise, the `logger_name` parameter can be any string specifying a logger name for all nodes in the provided cluster. | | `logger_level` | string | The level of severity to be assigned to the `logger_name`.
`logger_level` must be one of the following options:
`'EDEBUG' \| 'DEBUG' \| 'VERBOSE' \| 'INFO' \| 'WARN' \| 'ERROR'`
See the [Log Monitoring](/log-monitoring) page for more details on log levels.
To reset a logger to the default log level (`INFO`), use the `RESET` keyword. | **Examples** This example alters the log level at the cluster scope. This sets the query log level for all nodes in the cluster to `DEBUG`. ```sql SQL theme={null} ALTER CLUSTER cluster_name ALTER LOG LEVEL SET 'query' = 'DEBUG'; ``` This example resets the log level for the query logger for the entire cluster. ```sql SQL theme={null} ALTER CLUSTER cluster_name ALTER LOG LEVEL RESET 'query'; ``` #### ALTER CLUSTER RENAME TO Rename an existing cluster in the system. **Syntax** ```sql SQL theme={null} ALTER CLUSTER old_cluster_name RENAME TO new_cluster_name ``` | **Parameter** | **Data** **Type** | **Description** | | ------------------ | ----------------- | ------------------------------------------- | | `old_cluster_name` | string | The name of the specified cluster to alter. | | `new_cluster_name` | string | The new name to assign to the cluster. | **Example** This example renames an existing cluster named `lts-cluster-0` to `lts-cluster-1`. ```sql SQL theme={null} ALTER CLUSTER "lts-cluster-0" RENAME TO "lts-cluster-1"; ``` ## STORAGESPACE ### CREATE STORAGESPACE `CREATE STORAGESPACE` creates a new storage space. The name of the storage space must be distinct from the name of any existing storage spaces in the system. To create a storage space, you must possess the `CREATE STORAGESPACE` privilege for the current system. It is recommended that the total width of a production system be smaller than the number of Foundation Nodes in the cluster so that loading can continue even in the event of a node outage. **Syntax** ```sql SQL theme={null} CREATE STORAGESPACE [ IF NOT EXISTS ] storage_space_name [ WIDTH [=] width_integer], [ PARITY_WIDTH[=] pw_integer ] ``` | **Parameter** | **Data** **Type** | **Description** | | -------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `storage_space_name` | string | The name of the new storage space.
A storage space name must begin with a letter followed by letters, numbers, and underscores. | | `width_integer` | integer | The total number of nodes to use in each segment group as it is written to disk.
Width cannot be greater than the number of nodes in the storage cluster. | | `pw_integer` | integer | The number of parity coding bits to use for each segment group. This number must be less than `width`.
The specified storage space can still complete queries even if nodes are disabled as long as the number of disabled nodes does not exceed the `parity_width`.
For example, a storage space with a `width = 5 ` and a `parity_width = 2` can have up to two nodes become disabled, and the system still uses the three remaining nodes to execute queries. However, if a third node goes down, the system can no longer process query operations. | **Example** This example creates a new storage space named `ocient`. ```sql SQL theme={null} CREATE STORAGESPACE ocient WIDTH = 10, PARITY_WIDTH = 2; ``` ### DROP STORAGESPACE `DROP STORAGESPACE` removes an existing storage space, along with all associated tables and views. The SQL statement fails if one or more storage clusters are linked to the storage space. To remove a storage space, you must possess the `DROP STORAGESPACE` privilege for the storage space. Note that this action cannot be undone. **Syntax** ```sql SQL theme={null} DROP STORAGESPACE [ IF EXISTS ] storage_space_name [, ...] ``` | **Parameter** | **Data** **Type** | **Description** | | -------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `storage_space_name` | string | The name of a storage space to drop.
You can drop multiple storage spaces by specifying additional storage space names and separating each with commas. | **Example** Remove an existing storage space named `ocient_ss`. ```sql SQL theme={null} DROP STORAGESPACE ocient_ss; ``` Remove multiple storage spaces. ```sql SQL theme={null} DROP STORAGESPACE ocient_ss1, ocient_ss2; ``` ### ALTER STORAGESPACE `ALTER STORAGESPACE RENAME` renames an existing storage space. To rename a storage space, the logged-in user must be a system-level user. **Syntax** ```sql SQL theme={null} ALTER STORAGESPACE [ IF EXISTS ] old_storage_space_name RENAME TO new_storage_space_name ``` | **Parameter** | **Data** **Type** | **Description** | | ------------------------ | ----------------- | --------------------------------------------- | | `old_storage_space_name` | string | The name of a storage space to rename. | | `new_storage_space` | string | The new name for the specified storage space. | **Example** To rename an existing storage space named `teststoragespace` to `storage-space-1`. ```sql SQL theme={null} ALTER STORAGESPACE "teststoragespace" RENAME TO "storage-space-1"; ``` ## NODE ### DROP NODE `DROP NODE` removes an existing node from the system. A node cannot be removed from the system if it is still part of a cluster or one of the two remaining Administration Nodes. **Syntax** ```sql SQL theme={null} DROP NODE [ IF EXISTS ] node_name [, ...] ``` | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `node_name` | string | The name of a node to drop.
You can drop multiple nodes by specifying additional node names and separating each with commas. | **Example** Remove a node named `example_node`. ```sql SQL theme={null} DROP NODE example_node; ``` Remove multiple nodes. ```sql SQL theme={null} DROP NODE example_node1, example_node2; ``` ### ALTER NODE #### ALTER NODE RENAME TO `ALTER NODE RENAME` renames an existing node. To rename a node, you must be a system-level user. Renaming a node can cause a mismatch between the name of the node and the hostname of the node. Furthermore, you have to update users and scripts to use the new name of the node. Use this statement only to reverse these situations that can arise naturally (for example, you must replace a node with a node that has a different name). If you have a DNS configuration for the node, you must reconfigure the node address after renaming the node by using the [ALTER NODE SET ADDRESS](#alter-node-set-address) SQL statement. **Syntax** ```sql SQL theme={null} ALTER NODE [ IF EXISTS ] old_node_name RENAME TO new_node_name ``` | **Parameter** | **Data** **Type** | **Description** | | --------------- | ----------------- | ------------------------------------ | | `old_node_name` | string | The name of a node to rename. | | `new_node_name` | string | The new name for the specified node. | **Example** This example renames an existing node named `sql0` to `sql1`. ```sql SQL theme={null} ALTER NODE sql0 RENAME TO sql1; ``` #### ALTER NODE ADD ROLE `ALTER NODE ADD ROLE` adds a role to a node. Each node can have one or more roles that it performs. To add a role, you must be a system-level user. **Syntax** ```sql SQL theme={null} ALTER NODE [ IF EXISTS ] node_name ADD ROLE [ IF NOT EXISTS ] ::= { sql | streamloader | operatorvm | admin | health } ``` | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | ---------------------------- | | `node_name` | string | The name of a node to alter. | #### **Node Roles (**``**)** This table explains the supported roles for nodes. | **Role** | **Description** | **Notes** | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `streamloader` | The `streamloader` role allows the node to continually import a large amount of data into the database. | When adding the `streamloader` role, the node must be restarted in order for the change to take effect. | | `sql` | The `sql` role accepts a SQL statement, parses the statement, and compiles the statement. | You can remove this role only if there is at least one other SQL Node present. | | `operatorvm` | The `operatorvm` role manages database query plans and executes queries. | Added or removed automatically together with the `sql` or `lts` role. | | `lts` | The nodes with the `lts` role store data on disk in segments in a column-oriented fashion. The `lts` role is associated with the Foundation Nodes on the system. | Can only be added or removed by adding or removing the node as a participant in a Foundation cluster. | | `admin` | The `admin` role accepts the administrative protocol and updates system metadata. | Can only be removed if there are at least two other Administrative Nodes. | | `health` | A node with the `health` role is responsible for maintaining performance-related counters and statistics. | Present initially on every node. Cannot be removed. | **Example** Add SQL role to the node `sql1`. ```sql SQL theme={null} ALTER NODE sql1 ADD ROLE sql; ``` #### ALTER NODE REMOVE ROLE `ALTER NODE REMOVE ROLE` removes a role from a node. For a list of roles, see [Node Roles](#node-roles-\). Some node roles cannot be removed using an `ALTER NODE` SQL statement, including `health`, `lts`, and `operatorvm`. To remove a role, you must be a system-level user. The node must be restarted in order for this change to take effect. If you remove a SQL role from a SQL Node, the database removes the node from the connectivity pool. You must restart all SQL Nodes so that the other SQL Nodes understand that the node is no longer a SQL Node. When you remove the `admin` role, you must also remove the `/var/opt/ocient/metadataStorage.raft` file. **Syntax** ```sql SQL theme={null} ALTER NODE [ IF EXISTS ] node_name REMOVE ROLE [ IF EXISTS ] { sql | streamloader | admin } ``` | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | ---------------------------- | | `node_name` | string | The name of a node to alter. | **Example** This example removes the `sql` role from a node. ```sql SQL theme={null} ALTER NODE my_node REMOVE ROLE sql; ``` #### ALTER NODE ALTER CONFIG SET `ALTER NODE ALTER CONFIG SET` sets a configuration override for the configuration at the node scope. For an explanation of how configuration overrides work, see [Inspect the Current Configuration](/inspect-the-current-configuration). If the specified path to the configuration setting does not exist, it will be considered a new configuration and appended to the collection of existing configurations. To reset a configuration override for a specific key, use the `RESET` keyword. **Syntax** ```sql SQL theme={null} -- To set a new configuration override: ALTER NODE [ IF EXISTS ] node_name ALTER CONFIG SET string_definition [=] string_value [,...] -- To reset a configuration override: ALTER NODE [ IF EXISTS ] node_name ALTER CONFIG RESET string_definition [,...] ``` | **Parameter** | **Data** **Type** | **Description** | | ------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `node_name` | string | The name of the specified node to alter. | | `string_definition` | string | The name of a configuration parameter to alter.
Contact Ocient Support for details about altering configuration parameters. | | `string_value` | string | The new value to be set for the specified configuration parameter. | **Examples** This example sets the certificate name for this node to `cert_name.crt`. ```sql SQL theme={null} ALTER NODE node2 ALTER CONFIG SET 'certificateStore.cert' = 'cert_name.crt'; ``` This example resets the certificate configuration for the node `node2`. ```sql SQL theme={null} ALTER NODE node2 ALTER CONFIG RESET 'certificateStore.cert'; ``` This example alters the configuration at the node scope for node `node2` for service role `lts`. ```sql SQL theme={null} ALTER NODE node2 ALTER CONFIG SET 'lts.numLevels' = '3'; ``` #### ALTER NODE ALTER LOG LEVEL SET `ALTER NODE node_name ALTER LOG LEVEL SET` sets the log level for a particular logger on the specified node. For an explanation of how configuration overrides work, see [Inspect the Current Configuration](/inspect-the-current-configuration). The logger name specified can be any string specifying a logger name, or ALL for the root logging configuration. See the [Log Configuration](/log-monitoring#log-configuration) for more details. To reset a logger to the default log level (INFO), use the `RESET` keyword. **Syntax** ```sql SQL theme={null} -- To set a new configuration override: ALTER NODE [ IF EXISTS ] node_name ALTER LOG LEVEL SET { [,...] } ::= 'ALL' | logger_name [=] { 'EDEBUG' | 'DEBUG' | 'VERBOSE' | 'INFO' | 'WARN' | 'ERROR' } -- To reset a configuration override: ALTER NODE [ IF EXISTS ] node_name ALTER LOG LEVEL RESET [ logger_name [,...] ] ``` | **Parameter** | **Data** **Type** | **Description** | | --------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `node_name` | string | The name of the specified node to alter. | | `` | string | Use `ALL` for the root logging configuration.
Otherwise, the `logger_name` parameter can be any string specifying a logger name assigned to a severity level. See the [Log Configuration](/log-monitoring) page for more details.
To reset a logger to the default log level (`INFO`), use the `RESET` keyword. | **Examples** This example alters the log level at the node scope. The example sets the query log level for this particular node to `DEBUG`. ```sql SQL theme={null} ALTER NODE node_name ALTER LOG LEVEL SET 'query' = 'DEBUG'; ``` This example resets log level for the query logger on the specified node. ```sql SQL theme={null} ALTER NODE node_name ALTER LOG LEVEL RESET 'query'; ``` #### ALTER NODE ALTER METRIC LEVEL Alters the reporting level of various internal metrics on a specific node. For details about internal metrics, see [Statistics Monitoring](/statistics-monitoring). To set metrics reporting on all nodes across the system, see [ALTER SYSTEM ALTER METRIC LEVEL](#alter-system-alter-metric-level). **Syntax** ```sql SQL theme={null} ALTER NODE [ IF EXISTS ] node_name ALTER METRIC LEVEL { SET | RESET } ::= [ LIKE | REGEX ] metric_to_alter [ ,... ] [=] { 'INFO' | 'DEBUG' } ::= { [ LIKE | REGEX ] metric_to_reset [ ,... ] | ALL } ``` | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | ---------------------------- | | `node_name` | string | The name of a node to alter. | #### Set Metric Levels `` The `SET` SQL statement assigns one or more specified metrics to the specified reporting level. To set levels for a range of statistics, you can use wildcard characters with the `LIKE` keyword. Alternatively, you can use regular expressions with the `REGEX` keyword. | **Parameter** | **Data** **Type** | **Description** | | ----------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metric_to_alter` | string | The names of one or more metrics to alter for logging. Enclose any metric names in single quotes.
The supported logging level values for metrics are `INFO` and `DEBUG`. Either keyword must be in single quotes. | #### Reset Metric Levels `` The `RESET` SQL statement reverts one or more specified metrics to the default log level. You can use wildcard characters with the `LIKE` keyword. Alternatively, you can use regular expressions with the `REGEX` keyword. Use the `ALL` keyword to revert all metrics to the default log level. | **Parameter** | **Data** **Type** | **Description** | | ----------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `metric_to_reset` | string | The names of one or more metrics to reset by reverting their log level to the default.
To revert all metrics, use the `ALL` keyword. | **Examples** In this example, the `ALTER NODE` SQL statement sets the `resultCache.queries` metric on the node `oc1` to the `DEBUG` level. ```sql SQL theme={null} ALTER NODE oc1 ALTER METRIC LEVEL SET 'resultCache.queries' = 'DEBUG'; ``` This example uses the `LIKE` wildcard keyword to capture any metrics prefixed by `localStorageService.device.smart.` and assign them to the `INFO` level. ```sql SQL theme={null} ALTER NODE oc1 ALTER METRIC LEVEL SET LIKE 'localStorageService.device.smart.%' 'INFO'; ``` Similar to the previous example, this SQL statement uses regular expressions to capture any metrics prefixed by `localStorageService.device.smart.`. ```sql SQL theme={null} ALTER NODE oc1 ALTER METRIC LEVEL SET REGEX 'localStorageService.device.smart.*' 'INFO'; ``` This example alters the level of multiple metrics. ```sql SQL theme={null} ALTER NODE oc1 ALTER METRIC LEVEL SET 'resultCache.queries' = 'DEBUG', 'resultCache.data' = 'INFO', LIKE 'localStorageService.device.smart.%' 'INFO'; ``` This example resets the same metrics. ```sql SQL theme={null} ALTER NODE oc1 ALTER METRIC LEVEL RESET 'resultCache.queries', 'resultCache.data', LIKE 'localStorageService.device.smart.%'; ``` #### ALTER NODE SET ADDRESS `ALTER NODE node_name SET ADDRESS ip_address` changes the internal IP address or hostname for any node. You must restart the system after you change the address. If you change the address of an Administrator Node, then you must update the `bootstrap.conf` file for every node with the new address. For details about this file, see [Ocient System Bootstrapping](/ocient-system-bootstrapping). **Syntax** ```sql SQL theme={null} ALTER NODE [ IF EXISTS ] node_name SET ADDRESS node_address ``` | **Parameter** | **Data** **Type** | **Description** | | -------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `node_name` | string | The name of a node to alter. | | `node_address` | string | The address for a node.
This address can be an internal IP address (e.g., `111.11.11.111`) or a DNS hostname with the node address matching the node name (e.g., `sql1`). | **Examples** This example changes the IP address to `111.11.11.111` for the node named `my_node`. ```sql SQL theme={null} ALTER NODE my_node SET ADDRESS '111.11.11.111'; ``` This example changes the SQL Node `sql1` to use the DNS address `sql1`. ```sql SQL theme={null} ALTER NODE sql1 SET ADDRESS 'sql1'; ``` ## CONNECTIVITY POOL ### CREATE CONNECTIVITY\_POOL `CREATE CONNECTIVITY_POOL` creates a connectivity pool from participant SQL Nodes for a client connection. Connectivity pools enable you to change the IP address of the client connection. Client redirects happen only within connectivity pools. For details about connectivity pools and managing them, see [Manage the Network Configuration of an Ocient System](/manage-the-network-configuration-of-an-ocient-system). All SQL Nodes must be part of at least one connectivity pool. SQL Nodes only listen using the values set for the `listen_ip` and `listen_port` parameters. The database uses the `source_ip`, `source_port`, and `priority` parameters when multiple connectivity pools exist that a client can use for the connection. In this case, the database uses the connectivity pool with the highest priority. The database commits all changes made from these SQL statements after you restart all the SQL Nodes. For information on all connectivity pools on the system, query the system catalog tables for [Connectivity Pools](/system-catalog#connectivity-pools). To see the connectivity pool assignment for each node, query the system catalog table for [connectivity pool participants](/system-catalog#sys-connectivity_pool_participants). **Syntax** ```sql SQL theme={null} CREATE CONNECTIVITY_POOL [IF NOT EXISTS] pool_name ::= SOURCE_ADDRESS [=] source_ip | [ SOURCE_PORT [=] source_port ] | PRIORITY [=] priority | [ SSO INTEGRATION [=] sso_name ] | PARTICIPANTS ( (NODE [=] node_name LISTEN_ADDRESS [=] listen_ip LISTEN_PORT [=] listen_port ADVERTISED_ADDRESS [=] advertised_ip [ ADVERTISED_PORT [=] advertised_port ] [ OPENAPI_PORT [=] openapi_port ), ...) ``` | **Parameter** | **Data** **Type** | **Description** | | ----------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pool_name` | string | The name of the connectivity pool. | | `source_ip` | string | The source IP address, which is the IP address in CIDR notation of the client that connects to the SQL Node. | | `source_port` | integer | Optional.
The source port number is the port number of the client. This parameter defaults to none, which means any port number. | | `priority` | integer | The priority of the connection. A higher number indicates a higher priority. | | `sso_name` | string | The name of an SSO integration to use for the connectivity pool. For details, see [SSO INTEGRATION](#sso-integration). | | `node_name` | string | The name of the SQL Node. | | `listen_ip` | string | The IP address for listening.
ℹ️ Connectivity pools do not support dynamic IP addresses. If the IP address of your SQL Node changes, you must manually update it by using an [ALTER CONNECTIVITY\_POOL ALTER PARTICIPANT](#alter-connectivity_pool-alter-participant) SQL statement. | | `listen_port` | integer | The port number for listening. | | `advertised_ip` | string | The IP address to return to the client. | | `advertised_port` | integer | Optional.
The port number to return to the client. If you specify NULL for this parameter, then the database uses the value of the `listen_port` parameter. | | `openapi_port` | integer | The specified port for HTTP Query API-compliant specifications and endpoints. | **Examples** Create the connectivity pool named `test_pool` with source address `1.2.3.4/32` and priority `1` for one SQL Node named `sql0`. The node has the IP address `111.1.1.1` and port number `4050` for listening. Specify the local IP address to return to the client. ```sql SQL theme={null} CREATE CONNECTIVITY_POOL IF NOT EXISTS test_pool SOURCE_ADDRESS '1.2.3.4/32' PRIORITY 1 PARTICIPANTS ( (NODE sql0 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost')); ``` Create the connectivity pool named `test_pool` with source address `1.2.3.4/32`, port number `44`, and priority `1` for one SQL Node named `sql0`. The node has the IP address `111.1.1.1` and port number `4050` for listening. Specify the local IP address and port number `4050` to return to the client. ```sql SQL theme={null} CREATE CONNECTIVITY_POOL IF NOT EXISTS test_pool SOURCE_ADDRESS '1.2.3.4/32' SOURCE_PORT 44 PRIORITY 1 PARTICIPANTS ( (NODE sql0 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050)); ``` Create the connectivity pool named `test_pool` with source address `1.2.3.4/32`, port number `44`, priority `2`, and three participant SQL Nodes: `sql0`, `sql1`, and `sql2`. The nodes have the IP address `111.1.1.1` and port number `4050` for listening. Specify the local IP address and port number `4050` to return to the client. ```sql SQL theme={null} CREATE CONNECTIVITY_POOL IF NOT EXISTS test_pool SOURCE_ADDRESS '1.2.3.4/32' SOURCE_PORT 44 PRIORITY 2 PARTICIPANTS ( (NODE sql0 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050), (NODE sql1 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050), (NODE sql2 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050)); ``` ### DROP CONNECTIVITY\_POOL `DROP CONNECTIVITY_POOL` removes the specified connectivity pool. All nodes must belong to a connectivity pool. The execution of this SQL statement does not permit the removal of a pool such that a node becomes an orphan. **Syntax** ```sql SQL theme={null} DROP CONNECTIVITY_POOL [ IF EXISTS ] pool_name ``` | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | -------------------------------------------- | | `pool_name` | string | The name of the connectivity pool to remove. | **Example** Remove the connectivity pool named `test_pool`. ```sql SQL theme={null} DROP CONNECTIVITY_POOL test_pool; ``` ### ALTER CONNECTIVITY\_POOL #### ALTER CONNECTIVITY\_POOL SET `ALTER CONNECTIVITY_POOL SET` sets the metadata of a connectivity pool that includes the source IP address, source port number, priority, and node participants. **Syntax** ```sql SQL theme={null} ALTER CONNECTIVITY_POOL [ IF EXISTS ] pool_name SET ::= SOURCE_ADDRESS [=] source_ip | SOURCE_PORT [=] source_port | PRIORITY [=] priority | PARTICIPANTS( (NODE [=] node_name LISTEN_ADDRESS [=] listen_ip LISTEN_PORT [=] listen_port ADVERTISED_ADDRESS [=] advertised_ip [ADVERTISED_PORT [=] advertised_port] ), ...) ``` For parameter definitions, see [CREATE CONNECTIVITY POOL](#create-connectivity_pool). To directly edit the `LISTEN_ADDRESS`, `LISTEN_PORT`, `ADVERTISED_ADDRESS`, or `ADVERTISED_PORT` parameters, you must either remove the participants and add them back with new values for these parameters using the `ALTER CONNECTIVITY_POOL DROP PARTICIPANTS` and `ALTER CONNECTIVITY_POOL ADD PARTICIPANTS` statements, respectively, or use the `ALTER CONNECTIVITY_POOL SET PARTICIPANTS` statement. **Examples** This example sets the priority to `2` of the connectivity pool named `test_pool`. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL test_pool SET PRIORITY = 2; ``` This example sets the participants of the connectivity pool named `test_pool` to the list of participants with SQL Nodes `sql1` and `sql2`. When you execute this statement, the database resets the prior list of node participants to the new list. The nodes have the IP address `111.1.1.1` and port number `4050` for listening. Specify the local IP address and port number `4050` to return to the client. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL test_pool SET PARTICIPANTS( (NODE sql1 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050), (NODE sql2 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050)); ``` View the node identifiers of the participants of the `test_pool` connectivity pool by querying the node identifier `node_id` in the `sys.connectivity_pool_participants` and `sys.connectivity_pools` system catalog tables. ```sql SQL theme={null} SELECT cpp.node_id FROM sys.connectivity_pool_participants cpp INNER JOIN sys.connectivity_pools cp ON cpp.id = cp.id WHERE cp.name = 'test_pool'; ``` #### ALTER CONNECTIVITY\_POOL RENAME TO `ALTER CONNECTIVITY_POOL RENAME TO` SQL statement renames the existing connectivity pool. **Syntax** ```sql SQL theme={null} ALTER CONNECTIVITY_POOL [ IF EXISTS ] pool_name RENAME TO new_pool_name ``` | **Parameter** | **Data** **Type** | **Description** | | --------------- | ----------------- | ------------------------------------------- | | `pool_name` | string | The name of the existing connectivity pool. | | `new_pool_name` | string | The new name of the connectivity pool. | **Examples** Rename the `test_pool` connectivity pool to `new_test_pool`. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL test_pool RENAME TO new_test_pool; ``` #### ALTER CONNECTIVITY\_POOL ADD PARTICIPANTS `ALTER CONNECTIVITY_POOL ADD PARTICIPANTS` adds one or more nodes to the participant list of an existing connectivity pool. For information on all connectivity pools on the system, query the system catalog tables for [Connectivity Pools](/system-catalog). To see the connectivity pool assignment for each node, query the system catalog table for [connectivity pool participants](/system-catalog). **Syntax** ```sql SQL theme={null} ALTER CONNECTIVITY_POOL [ IF EXISTS ] pool_name ADD PARTICIPANTS( (NODE [=] node_name LISTEN_ADDRESS [=] listen_ip LISTEN_PORT [=] listen_port ADVERTISED_ADDRESS [=] advertised_ip [ADVERTISED_PORT [=] advertised_port] ), ...) ``` **Examples** Add one node `sql5` to the participant list of the connectivity pool `test_pool`. The node has the IP address `111.1.1.1` and port number `4050` for listening. Specify the local IP address and port number `4050` to return to the client. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL test_pool ADD PARTICIPANTS( NODE sql5 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050); ``` Add two nodes `sql6` and `sql7` to the participants list of the connectivity pool `test_pool`. The nodes have the IP address `111.1.1.1` and port number `4050` for listening. Specify the local IP address and port number `4050` to return to the client. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL test_pool ADD PARTICIPANTS( (NODE sql6 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050), (NODE sql7 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050)); ``` #### ALTER CONNECTIVITY\_POOL ALTER PARTICIPANT `ALTER CONNECTIVITY_POOL ALTER PARTICIPANT` modifies the configurations of one SQL Node assigned to the existing connectivity pool. This SQL statement can alter only one configuration parameter of a participant node at a time. For information on all connectivity pools on the system, query the system catalog tables for [Connectivity Pools](/system-catalog). To see the connectivity pool assignment for each node, query the system catalog table for [connectivity pool participants](/system-catalog). **Syntax** ```sql SQL theme={null} ALTER CONNECTIVITY_POOL [ IF EXISTS ] pool_name ALTER PARTICIPANT node_name SET ::= { LISTEN_ADDRESS [=] listen_ip | LISTEN_PORT [=] listen_port | ADVERTISED_ADDRESS [=] advertised_ip | ADVERTISED_PORT [=] advertised_port | OPENAPI_PORT [=] openapi_port } ``` | **Parameter** | **Data** **Type** | **Description** | | ----------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `pool_name` | string | The name of the connectivity pool. | | `node_name` | string | The name of the SQL Node to alter. | | `listen_ip` | string | The IP address for listening. | | `listen_port` | numeric | The port number for listening. | | `advertised_ip` | string | The IP address to return to the client. | | `advertised_port` | numeric | The port number to return to the client. If you specify NULL for this parameter, then the database uses the value of the `listen_port` parameter. | | `openapi_port` | numeric | The specified port for HTTP Query API-compliant specifications and endpoints. | **Example** Modify the SQL Node `sql0` to use the new `listen_port` value `4051`. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL cp_test ALTER PARTICIPANT sql0 SET listen_port 4051; ``` #### ALTER CONNECTIVITY\_POOL DROP PARTICIPANTS `ALTER CONNECTIVITY_POOL DROP PARTICIPANTS` removes one or more nodes from the participant list of an existing connectivity pool. If you remove the last node of a connectivity pool, the Ocient System automatically removes the connectivity pool. For information on all connectivity pools on the system, query the system catalog tables for [Connectivity Pools](/system-catalog). To see the connectivity pool assignment for each node, query the system catalog table for [connectivity pool participants](/system-catalog). All nodes must belong to a connectivity pool. The execution of this SQL statement does not permit the removal of a node such that it becomes an orphan. **Syntax** ```sql SQL theme={null} ALTER CONNECTIVITY_POOL [ IF EXISTS ] pool_name DROP PARTICIPANTS (node_name, ...) ``` **Examples** Remove one node `sql5` from the connectivity pool `test_pool`. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL IF EXISTS test_pool DROP PARTICIPANTS sql5; ``` Remove nodes `sql6` and `sql7` from the connectivity pool `test_pool`. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL test_pool DROP PARTICIPANTS (sql6, sql7); ``` #### ALTER CONNECTIVITY\_POOL SET SSO INTEGRATION `ALTER CONNECTIVITY_POOL SET SSO INTEGRATION` assigns a connectivity pool to use a specific SSO integration. Connectivity pools look for the specified SSO integration name for each database you are trying to connect to. If the database does not support that SSO integration name, it uses the default SSO. **Syntax** ```sql SQL theme={null} ALTER CONNECTIVITY_POOL [ IF EXISTS ] pool_name SET SSO INTEGRATION sso_name ``` | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | ------------------------------------------- | | `pool_name` | string | The name of the existing connectivity pool. | | `sso_name` | string | The new name of the SSO integration. | **Example** ```sql SQL theme={null} ALTER CONNECTIVITY_POOL cp_test SET SSO INTEGRATION sso_test; ``` #### ALTER CONNECTIVITY\_POOL REMOVE SSO INTEGRATION `ALTER CONNECTIVITY_POOL REMOVE SSO INTEGRATION` removes an assigned SSO integration. **Syntax** ```sql SQL theme={null} ALTER CONNECTIVITY_POOL [ IF EXISTS ] pool_name REMOVE SSO INTEGRATION sso_name ``` | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | ------------------------------------------- | | `pool_name` | string | The name of the existing connectivity pool. | | `sso_name` | string | The name of the existing SSO integration. | **Example** This example removes the SSO integration `sso_test` from the connectivity pool `cp_test`. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL cp_test REMOVE SSO INTEGRATION sso_test; ``` ## SSO INTEGRATION ### CREATE SSO INTEGRATION Creates a new SSO integration protocol, which can connect to databases as a database integration (see [ALTER DATABASE SET SSO INTEGRATION](/databases#alter-database-set-sso-integration) ) or connect using a connectivity pool (see [ALTER CONNECTIVITY\_POOL SET SSO INTEGRATION](#alter-connectivity_pool-set-sso-integration)). **Syntax** ```sql SQL theme={null} CREATE SSO INTEGRATION [IF NOT EXISTS ] sso_name PROTOCOL sso_protocol [, ... ] ::= -- literal or string value property_name = value | -- list of values property_name = [ value [, ...] ] | -- map of values property_name = { key = value [, ...] } ``` | **Parameter** | **Type** | **Description** | | --------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sso_name` | string | The identifier of the SSO integration to create. | | `sso_protocol` | string | Supported values are: `oidc`, `openid`, and `openid_connect`.
All these values are aliases of each other and use the OAuth 2.0 Authorization. | | `property_name` | string | One or more properties to include in the SSO integration. Each SSO property must be specified either as a literal, list, or map. See the SSO Properties table for the requirements for each supported SSO parameter.
SSO properties also have these rules:
You can place any string key or value between double quotations (e.g., `"default_group"`).
String values that contain characters other than `[a-zA-z] \| [0-9] \| '_' ` require double quotations (e.g., `"this.is.a.complex-$tring"`).
Using a NULL value clears the existing configuration of a list or map property (e.g., `user_claim_ids = NULL`). | #### SSO Properties | **Name** | **Required** | **Value Type** | **Default Value** | **Description** | | -------------------------------- | ------------ | ------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `issuer` | Yes | string | None | The [complete URL](https://openid.net/specs/openid-connect-core-1_0.html#Terminology) for the OAuth 2.0 and OpenID Connect Authorization Server. This property value is the expected `\"iss\"` claim in access tokens validated by the database. | | `client_id` | Yes | string | None | The [client identifier](https://openid.net/specs/openid-connect-core-1_0.html#Terminology) as registered with the OpenID Provider. | | `client_secret` | Yes | string | None | The [client secret](https://openid.net/specs/openid-connect-core-1_0.html#IDTokenValidation) as registered with the OpenID Provider. This property is required for some SSO workflows. | | `default_group` | Yes | string | None | The group users are assigned if no group is specified in the `group_claim_mappings` property for the OpenID Provider. | | `redirect_host` | No | string | Depends on the connector (pyocient or JDBC). | Specifies the host name during SSO redirection. | | `redirect_ssl` | No | Boolean | Depends on the connector (pyocient or JDBC). | Set to `true` to enable SSL callback during SSO redirection (i.e., redirect uses `https`).
Set to `false` to disable (i.e., redirect uses `http`). | | `disabled` | No | Boolean | `false` | Set to `true` to disable the OIDC integration for maintenance temporarily.
⚠️ If you set this property to `true`, all authentication requests using this connection fail. | | `enable_id_token_authentication` | No | Boolean | `false` | Set this property to `true` if the identifier token also contains the authorization token.
In most circumstances, this option is necessary only for machine-to-machine connections without user interaction, such as a server using a script to connect to an Ocient System. | | `user_id_claims` | No | list of strings (e.g., value \[, ...] | \["email"] | Set the identifier token claims used to identify users in audit trails. If you do not set a value, the system uses the `"email"` claim if it is present, otherwise it uses `["iss", "sub"]`. | | `additional_scopes` | No | list of strings (e.g., value \[, ...] | \[] | Specifies additional scopes to request when executing the Authorization Code Flow. | | `additional_audiences` | No | list of strings (e.g., value \[, ...] | \[] | Specifies additional audiences to accept when validating tokens. This property is useful for authorization servers without the token exchange capability.
Each additional audience is a case-sensitive URL from a provider, similar to the `issuer` value. | | `groups_claim_ids` | No | list of strings (e.g., value \[, ...] | \[] | Specifies the token claims that can be used to map the user to a Database group. If you specify the `groups_claim_mappings` property, you must also specify the `groups_claim_ids` property. | | `groups_claim_mappings` | No | map of strings (e.g., key = value \[, ...]) | Specifies mappings from the Provider group to the Database group. | | | `roles_claim_ids` | No | list of strings (e.g., value \[, ...] | \[] | Specifies the token claims that can be used to map the user to a Database role. If you specify the `roles_claim_mappings` property, you must also specify the `roles_claim_ids` property. | | `roles_claim_mappings` | No | map of strings (e.g., key = value \[, ...]) | Specifies mappings from the Provider role to the Database role. | | | `allowed_groups` | No | list of strings (e.g., value \[, ...] | \[] | Specifies a list of external identity provider groups that are permitted to authenticate through this SSO integration. If you specify the `allowed_groups` property, only users who are members of the specified identify provider groups can access the Ocient System.

If this property is empty or you do not specify it, then group-based filtering is disabled, and the system allows all authenticated users from the provider unless they are explicitly blocked. Group names must match exactly as they appear in the claims of the identify provider group. | | `allowed_roles` | No | list of strings (e.g., value \[, ...] | \[] | Defines a list of external identity provider roles that are authorized to access the Ocient System through this SSO integration. Only users assigned to the specified roles in the external provider can authenticate.

If this property is empty or you do not specify it, role-based access control is disabled for this integration. Role names must correspond exactly to the role claims provided by the identity provider in the authentication response. | | `blocked_groups` | No | list of strings (e.g., value \[, ...] | \[] | Specifies a list of external identity provider groups that are explicitly denied access through this SSO integration. The system blocks users who are members of any specified group in the external identity provider from authenticating, regardless of other permissions they have. Setting this property takes precedence over the `allowed_groups` property if a user belongs to both an allowed group and a blocked group. Group names must match exactly as they appear in the claims of the identify provider group. | | `blocked_roles` | No | list of strings (e.g., value \[, ...] | \[] | Defines a list of external identity provider roles that are explicitly prohibited from accessing the Ocient System through this SSO integration. The system denies access to users assigned to any of the specified roles, overriding any other access permissions. Setting this property takes precedence over the
`allowed_roles` property in cases where a user is in both an allowed role and a blocked role. Role names must correspond exactly to the role claims of the identity provider group. | | `allow_offline_access` | No | Boolean | `false` | When the `allow_offline_access` property is `true`, Ocient requests offline access from the OpenID Connect identity provider by including `access_type=offline` and `prompt=consent` in the authorization URL. | **Example** This example creates an SSO integration protocol with the specified OpenID issuer `"https://accounts.google.com"`, client identifier, and default group. Each of these parameters is required to create a new SSO integration. The example assumes preset values for the client identifier and default group. ```sql SQL theme={null} CREATE SSO INTEGRATION sso_test PROTOCOL oidc issuer = "https://accounts.google.com", client_id = example_database_app_id, default_group = example_database_group; ``` ### DROP SSO INTEGRATION Drops an SSO integration protocol. **Syntax** ```sql SQL theme={null} DROP SSO INTEGRATION [ IF EXISTS ] sso_name ``` | **Parameter** | **Type** | **Description** | | ------------- | -------- | ---------------------------------------------- | | `sso_name` | string | The identifier of the SSO integration to drop. | **Example** This SQL statement drops the SSO integration protocol `sso_test`. ```sql SQL theme={null} DROP SSO INTEGRATION sso_test; ``` ### ALTER SSO INTEGRATION Alters an SSO integration protocol by renaming it or modifying its properties. ```sql SQL theme={null} ALTER SSO INTEGRATION sso_name [ IF EXISTS ] { RENAME TO new_name | SET [, ... ] } ::= -- literal or string value property_name = value | -- list of values property_name = [ value [, ...] ] | -- map of values property_name = { key = value [, ...] } ``` | **Parameter** | **Type** | **Description** | | --------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sso_name` | string | The identifier of the SSO integration to alter. | | `new_name` | string | The new name of the SSO integration. | | `property_name` | string | One or more properties to include in the SSO integration. Each SSO property must be specified either as a literal, list, or map. See the [SSO Properties](#sso-properties) table to see the requirements for each supported SSO parameter.
SSO properties also have these rules:
You can place any string key or value between double quotations (e.g., `"default_group"`).
String values that contain characters other than `[a-zA-z] \| [0-9] \| '_' ` require double quotations (e.g., `"this.is.a.complex-$tring"`).
Using a NULL value clears the existing configuration of a list or map property (e.g., `user_claim_ids = NULL`). | **Examples** **Alter SSO Integration Properties** This example alters the SSO integration to use different properties, changing the OpenID client identifier and the default group to `"group2"`. The example assumes a preset value for the client identifier. ```sql SQL theme={null} ALTER SSO INTEGRATION sso_test SET client_id = different_database_app_id, default_group = "group2"; ``` **Rename an SSO Integration** This example renames the SSO integration to `sso_test_2`. ```sql SQL theme={null} ALTER SSO INTEGRATION sso_test RENAME TO sso_test_2; ``` ## SYSTEM ### ALTER SYSTEM ALTER CONFIG SET `ALTER SYSTEM ALTER CONFIG SET` sets an override for the configuration at the system scope. For an explanation of how configuration overrides work, see [Inspect the Current Configuration](/inspect-the-current-configuration). If the specified path to the configuration setting does not exist, it is considered a new configuration and appended to the collection of existing configurations. To reset a configuration override for a specific parameter, use the `RESET` keyword. This configuration override is set in the base node configuration by default. To apply the override to a specific service role, prefix the key with the service role name. **Syntax** ```sql SQL theme={null} -- To set a new configuration override: ALTER SYSTEM ALTER CONFIG SET parameter_name [=] parameter_value [,...] -- To reset a configuration override: ALTER SYSTEM ALTER CONFIG RESET [ parameter_name [,...] ] ``` | **Parameter** | **Data** **Type** | **Description** | | ----------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `parameter_name` | string | The name of a configuration parameter to alter. This parameter should be a string literal (i.e., enclosed in single quotes).
Contact Ocient Support for details about altering configuration parameters. | | `parameter_value` | any | The new value to be set for the specified configuration parameter.
The data type of this value depends on the configuration parameter. This can be a string literal that can be cast to the data type of the configuration parameter. For example, the string literal `'true'` is equivalent to the Boolean value `true`. | **Examples** This example alters the base configuration of the system scope. The example sets the certificate name for each node in the entire system to `cert_name.crt`. ```sql SQL theme={null} ALTER SYSTEM ALTER CONFIG SET 'certificateStore.cert' = 'cert_name.crt'; ``` This example resets the certificate configuration for the entire system. ```sql SQL theme={null} ALTER SYSTEM ALTER CONFIG RESET 'certificateStore.cert'; ``` This example alters the configuration for the service role `sql`. The `purgeCacheFrequency` setting determines the frequency in seconds to purge the result-set cache of stale values. ```sql SQL theme={null} ALTER SYSTEM ALTER CONFIG SET 'sql.purgeCacheFrequency' = 600; ``` #### ALTER SYSTEM ALTER LOG LEVEL `ALTER SYSTEM ALTER LOG LEVEL SET` sets the log level for a particular logger for all nodes in the system. For an explanation of how configuration overrides work, see [Inspect the Current Configuration](/inspect-the-current-configuration). The logger name specified can be any string specifying a logger name, or `ALL` for the root logging config. See the [Log Configuration](/log-monitoring) for more details. **Syntax** ```sql SQL theme={null} -- To set a new configuration override: ALTER SYSTEM ALTER LOG LEVEL SET { [,...] } ::= 'ALL' | logger_name [=] { 'EDEBUG' | 'DEBUG' | 'VERBOSE' | 'INFO' | 'WARN' | 'ERROR' } -- To reset a configuration override: ALTER SYSTEM ALTER LOG LEVEL RESET [logger_name [, ...]] ``` | **Parameter** | **Data** **Type** | **Description** | | --------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | string | Use `ALL` for the root logging config.
Otherwise, the `logger_name` parameter can be any string specifying a logger name assigned to a severity level. See the [Log Configuration](/log-monitoring) page for more details.
To reset a logger to the default log level (`INFO`), use the `RESET` keyword. | **Example** This example alters the log level at the system scope. ```sql SQL theme={null} ALTER SYSTEM ALTER LOG LEVEL SET 'ALL' = 'DEBUG'; ``` ### ALTER SYSTEM RENAME TO `ALTER SYSTEM RENAME TO` sets the system name. The system name appears in the `sys.system_information` system catalog table. You can also see the current system name by using the [CURRENT\_SYSTEM](/other-functions-and-expressions#current_system) function. **Syntax** ```sql SQL theme={null} ALTER SYSTEM RENAME TO system_name ``` | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | ------------------------------ | | `system_name` | string | The name of the Ocient System. | **Example** ```sql SQL theme={null} ALTER SYSTEM RENAME TO "productionSystem"; ``` ### ALTER SYSTEM ALTER METRIC LEVEL Alters the reporting level of various internal metrics across the system, including all nodes. For details about internal metrics, see [Statistics Monitoring](/statistics-monitoring). To set metrics reporting for individual nodes, see [ALTER NODE ALTER METRIC LEVEL](#alter-node-alter-metric-level). **Syntax** ```sql SQL theme={null} ALTER SYSTEM ALTER METRIC LEVEL { SET | RESET } ::= [ LIKE | REGEX ] metric_to_alter [ ,... ] [=] { 'INFO' | 'DEBUG' } ::= { [ LIKE | REGEX ] metric_to_reset [ ,... ] | ALL } ``` #### Set Metric Levels `` The `SET` SQL statement assigns one or more specified metrics to the specified reporting level. You can assign metrics to `INFO` or `DEBUG` levels. To set levels for a range of statistics, you can use wildcard characters with the `LIKE` keyword. Alternatively, you can use regular expressions with the `REGEX` keyword. | **Parameter** | **Data** **Type** | **Description** | | ----------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metric_to_alter` | string | The names of one or more metrics to alter for logging. Enclose any metric names in single quotes.
The supported logging levels for metrics are `INFO` and `DEBUG`. Either keyword must be in single quotes. | #### Reset Metric Levels `` The `RESET` SQL statement reverts one or more specified metrics to the default reporting level. You can use wildcard characters with the `LIKE` keyword. Alternatively, you can use regular expressions with the `REGEX` keyword. Use the `ALL` keyword to revert all metrics to the default level. | **Parameter** | **Data** **Type** | **Description** | | ----------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `metric_to_reset` | string | The names of one or more metrics to reset by reverting their level to the default.
To revert all metrics, use the `ALL` keyword. | **Examples** In this example, the `ALTER SYSTEM` SQL statement sets the `resultCache.queries` metric to the `DEBUG` level. ```sql SQL theme={null} ALTER SYSTEM ALTER METRIC LEVEL SET 'resultCache.queries' = 'DEBUG'; ``` This example uses the `LIKE` wildcard to capture any metrics prefixed by `localStorageService.device.smart.` and assign them to the `INFO` level. ```sql SQL theme={null} ALTER SYSTEM ALTER METRIC LEVEL SET LIKE 'localStorageService.device.smart.%' 'INFO'; ``` Similar to the previous example, this SQL statement uses regular expressions to capture any metrics prefixed by `localStorageService.device.smart.`. ```sql SQL theme={null} ALTER SYSTEM ALTER METRIC LEVEL SET REGEX 'localStorageService.device.smart.*' 'INFO'; ``` This example alters the level of multiple metrics. ```sql SQL theme={null} ALTER SYSTEM ALTER METRIC LEVEL SET 'resultCache.queries' = 'DEBUG', 'resultCache.data' = 'INFO', LIKE 'localStorageService.device.smart.%' 'INFO'; ``` This example resets the same metrics. ```sql SQL theme={null} ALTER SYSTEM ALTER METRIC LEVEL RESET 'resultCache.queries', 'resultCache.data', LIKE 'localStorageService.device.smart.%'; ``` ### ALTER SYSTEM ALTER SECURITY Set security settings using the `ALTER SYSTEM ALTER SECURITY` SQL statement. Replace `` with the security setting and `` with the value. You can inspect the current values of system configuration settings using the `sys.config` and `sys.node_config` system catalog tables. **Syntax** ```sql SQL theme={null} ALTER SYSTEM ALTER SECURITY [=] ``` | **Parameter** | **Data** **Type** | **Description** | | ------------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `security_setting` | string | The security setting with values:
\* `password_minimum_length`
\* `password_complexity_level`
\* `password_no_repeat_count`
\* `password_lifetime_days`
\* `password_invalid_attempt_limit`
For details about these values, see [Database Password Security Settings](/database-password-security-settings). | | `value` | numeric | An integer to represent one of the security settings. For details about this value, see [Database Password Security Settings](/database-password-security-settings). | **Examples** **Set a Security Setting for All Databases** Set the password lifetime to 30 days for the entire system. ```sql SQL theme={null} ALTER SYSTEM ALTER SECURITY password_lifetime_days = 30; ``` **Set Multiple Security Settings** Set multiple security settings in a single SQL statement. In this case, set the password lifetime to 30 days and the minimum password character length to 12. As with other similar SQL statements, the `=` character is optional. ```sql SQL theme={null} ALTER SYSTEM ALTER SECURITY password_lifetime_days 30, password_minimum_length 12; ``` ### ALTER SYSTEM SET DEFAULT STORAGESPACE Sets the specified storage space as the default for creating new tables. `CREATE TABLE` SQL statements automatically use the default storage space unless you specify a different storage space (see [Table Options](/tables#create-option-\)). After you create a table, you cannot alter its assigned storage space. Statements such as `DROP STORAGESPACE` and `ALTER CLUSTER REMOVE STORAGESPACE` throw an error if they attempt to drop the default storage space of the system. To remove the default storage space, use the optional `RESET` keyword. The `RESET` keyword alters the system to have no default storage space. Until you set a storage space as the new default, `CREATE TABLE` statements throw an error if they do not specify a storage space for the new table. **Required Privileges** To use this SQL statement, you must have the `ALTER` privilege for the system. For details about privileges, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). **Syntax** ```sql SQL theme={null} ALTER SYSTEM { SET | RESET } DEFAULT STORAGESPACE [ storage_space_name ] ``` | **Parameter** | **Data** **Type** | **Description** | | -------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `storage_space_name` | string | The name of a storage space to set as the system default. This parameter is required when you use the `SET` keyword to define a default storage space.

Do not include this parameter when you use the `RESET` keyword to remove the default storage space. | **Examples** **Set a Default Storage Space** This example sets `ss_0` as the system default storage space. ```sql SQL theme={null} ALTER SYSTEM SET DEFAULT STORAGESPACE ss_0; ``` **Remove the Default Storage Space** This example removes any default storage space. ```sql SQL theme={null} ALTER SYSTEM RESET DEFAULT STORAGESPACE; ``` ## Related Links [Other Functions and Expressions](/other-functions-and-expressions) [CREATE TABLE SQL Statement Examples](/create-table-sql-statement-examples) [Database Password Security Settings](/database-password-security-settings) # Clustering Analysis and Dimensionality Reduction Source: https://docs.ocient.com/clustering-analysis-and-dimensionality-reduction Apply clustering and dimensionality reduction in Ocient with OcientML using k-means, PCA, and related algorithms for unsupervised exploration of data. This tutorial uses examples to explain the clustering and dimensionality reduction capabilities in . ## Clustering Models Clustering models bear similarities to classification, but they use unsupervised learning, which means they do not use any class labels. Instead, the algorithm tries to identify groupings on its own by finding clusters of data that seem closer to each other and farther away from other clusters. These models generate an integer label, but its ordering is arbitrary. The clustering models require you to specify upfront the number of clusters. ### K-Means Clustering K-means is by far the most well-known clustering algorithm because it is simple and fast. K-means performs particularly well if you can scale your features so clusters are roughly circular and equal in size. These examples demonstrate the k-means model. The examples do not use a data set that is primed for optimal performance of the model, such that they show the shortcomings of the k-means model and how it can perform relatively well even with sub-optimal data. The examples use a data set of three-dimensional points. ```sql SQL theme={null} SELECT x, y, z FROM mldemo.clusters_3d LIMIT 10; x y z ------------------------------------------------------------------ -2.332818550827427 2.1009728159912044 -0.7199553405333495 0.27417054580238553 -0.29545777905624226 0.6797340230215586 -2.8608875606526794 2.8532647291218884 -0.7296054608643955 -4.9539478834915345 -4.987358010702081 -0.818608017693332 0.7043958454569171 -0.7209720558364705 -0.7879961167668906 4.949298606341526 4.956911476243746 0.6015700102502752 2.245935331812275 -2.158347482865157 -1.0786578947921444 -1.3725012757407247 1.5652243270484643 -2.5951429274728794 4.516051945094492 4.560705560578962 -0.1569348000058415 5.140105808358035 -4.9583101530224 -1.0064835641873153 Fetched 10 rows ``` This example shows a k-means model created over this data. You can specify multiple options, but the only one that is required is the `k` value, which represents the number of clusters. ```sql SQL theme={null} CREATE MLMODEL kmeans1 TYPE KMEANS ON ( SELECT x, y, z FROM mldemo.clusters_3d ) options('k'->'3'); Modified 0 rows ``` This data set has labels, but they are hidden from the model to allow for unsupervised learning. These example queries compare how well the unsupervised k-means clustering performs against the actual correct classes and labels. Before you can do that, you must figure out which cluster numbers correspond to which labels. ```sql SQL theme={null} SELECT COUNT(*), cluster_num FROM ( SELECT kmeans1(x, y, z) AS cluster_num FROM mldemo.clusters_3d WHERE label = 'LEFT' ) GROUP BY cluster_num; count(*) cluster_num -------------------------------- 3165 0 146329 1 Fetched 2 rows SELECT COUNT(*), cluster_num FROM ( SELECT kmeans1(x, y, z) AS cluster_num FROM mldemo.clusters_3d WHERE label = 'RIGHT' ) GROUP BY cluster_num; count(*) cluster_num -------------------------------- 3877 0 145761 2 Fetched 2 rows SELECT COUNT(*), cluster_num FROM ( SELECT kmeans1(x, y, z) AS cluster_num FROM mldemo.clusters_3d WHERE label = 'CENTER' ) GROUP BY cluster_num; count(*) cluster_num -------------------------------- 700868 0 Fetched 1 row ``` The results indicate that cluster `0` is the `CENTER` class, cluster `1` is `LEFT`, and cluster `2` is `RIGHT`. With that information, you can compute the overall accuracy. ```sql SQL theme={null} SELECT count(*) / 1000000.0 FROM mldemo.clusters_3d WHERE ( kmeans1(x, y, z) = 0 AND label = 'CENTER' ) OR ( kmeans1(x, y, z) = 1 AND label = 'LEFT' ) OR ( kmeans1(x, y, z) = 2 AND label = 'RIGHT' ); (_count(*)_0)/((1000000.0)) ---------------------------- 0.992958 Fetched 1 row ``` The accuracy is very good. This calculation indicates that the model never miscategorized `CENTER`, but there were some wrong classifications for the `LEFT` and `RIGHT` labels, although they were rare. For details, see [K-Means Clustering](/clustering-and-dimension-reduction-models#k-means-clustering). ### Gaussian Mixture Models Gaussian mixture models (GMMs) also perform clustering, however they use a significantly more complex algorithm. GMMs can handle several things that k-means cannot, such as: * GMMs can handle clusters that are not circular, i.e., they have different variances in different directions. * GMMs can handle clusters that have an arbitrary rotation, i.e., they can have covariances. * GMMs can handle the fact that all clusters might not be equally as likely, i.e. if a point is located right between two clusters, then it is more likely to be the one more common in the training data. * GMMs can show the probability of a new point belonging to each cluster rather than outputting a single cluster value. GMMs operate by finding a weighted mixture of k multi-variate Gaussians that is most likely to represent the population from which the data was sampled. Each Gaussian in the mix has a mean vector, which represents the center of each cluster. After you create the model, it is simple to determine the cluster that a point most likely belongs to. The examples demonstrate how to find the highest probability cluster. This example makes a model over the same data set. This model requires the `numDistributions` option, which represents the number of clusters. ```sql SQL theme={null} CREATE MLMODEL gmm TYPE GAUSSIAN MIXTURE MODEL ON ( SELECT x, y, z FROM mldemo.clusters_3d ) options('metrics'->'true', 'numDistributions'->'3'); Modified 0 rows ``` This query executes the GMM model. ```sql SQL theme={null} SELECT gmm(0,0,0) AS class_probabilities; class_probabilities -------------------------------------------------------------------------------- [[2.0445296213545875E-4, 0.999582295498657, 2.1325153920763012E-4]] Fetched 1 row ``` The results are very different than executing a k-means model because these are probabilities. In this case, the output means the probability of `(0,0,0)` being the second class is very high (greater than 99.9 percent), while the probability of it being the other classes is essentially zero. To find the most likely class, use the `VECTOR_ARGMAX` function, which returns a 1-based class index. ```sql SQL theme={null} SELECT VECTOR_ARGMAX(gmm(0,0,0)) AS class_probabilities; class_probabilities -------------------- 1 Fetched 1 row ``` You can write a query to assess the accuracy of this model. But first, you must figure out the association between classes and labels. ```sql SQL theme={null} SELECT COUNT(*), cluster_num FROM ( SELECT VECTOR_ARGMAX(gmm(x, y, z)) AS cluster_num FROM mldemo.clusters_3d WHERE label = 'LEFT' ) GROUP BY cluster_num; count(*) cluster_num ---------------------------------------- 149483 3 2 1 9 2 Fetched 3 rows SELECT COUNT(*), cluster_num FROM ( SELECT VECTOR_ARGMAX(gmm(x, y, z)) AS cluster_num FROM mldemo.clusters_3d WHERE label = 'CENTER' ) GROUP BY cluster_num; count(*) cluster_num ---------------------------------------- 700868 1 Fetched 1 row SELECT COUNT(*), cluster_num FROM ( SELECT VECTOR_ARGMAX(gmm(x, y, z)) AS cluster_num FROM mldemo.clusters_3d WHERE label = 'RIGHT' ) GROUP BY cluster_num; count(*) cluster_num ---------------------------------------- 6 2 3 3 149629 1 Fetched 3 rows ``` This model misclassified 20 rows, which is far better than the 7,042 rows the k-means model misclassified. This is a direct result of the additional complexity of the GMM. GMM models can handle much more complex situations than k-means models, but this comes at the cost of more time training and executing the model. The time a GMM model takes compared to a k-means model depends on the number of clusters. For details, see [Gaussian Mixture](/clustering-and-dimension-reduction-models#gaussian-mixture). ## Dimensionality Reduction Dimensionality reduction algorithms reduce the number of input features while still keeping as much of the meaningful properties of the data as possible. Models are quicker to build, and often higher quality after a dimensionality reduction model reduces the number of input features. A common first step in an analysis is to use dimensionality reduction to simplify the data. See these examples that demonstrate how to use dimensionality reduction as a preprocessing step before using other model types. ### Principal Component Analysis Principal component analysis (PCA) is an unsupervised algorithm that only operates on the inputs, and does not understand what the data is being used for. It is also a linear dimensionality reduction algorithm, meaning that the new features it generates are linear combinations of existing features. PCA generates as many new features as there are input features. So by itself, it is not reducing the number of dimensions. However, PCA creates new features that try to maximize variance and sorts the new features in terms of the amount of variance they contain. The system catalog tables provide information on how much variance is contained by the new features. You can use this information to find how many trailing new features to drop. This PCA tutorial uses a new data set. Start with a regression problem that is trying to find the best-fit polynomial for `f(x1, x2, x3) = y`. The tutorial starts with a [Polynomial Regression](/regression-models#polynomial-regression) model to see how well it does using three input features, and then it uses PCA to reduce the number of variables without losing accuracy. This example shows the Polynomial Regression model. ```sql SQL theme={null} CREATE MLMODEL poly TYPE POLYNOMIAL REGRESSION ON ( SELECT x1, x2, x3, y FROM mldemo.pca_poly ) options('order'->'2', 'metrics'->'true'); Modified 0 rows ``` The system catalog table shows that the model fits the data perfectly. ```sql SQL theme={null} SELECT coefficient_of_determination FROM sys.machine_learning_models a, sys.polynomial_regression_models b WHERE a.id = b.machine_learning_model_id AND name = 'poly'; coefficient_of_determination ----------------------------- 1.0 Fetched 1 row ``` In this instance, PCA can reduce the three independent variables to two. The first step is to build a PCA model over the input features. ```sql SQL theme={null} CREATE MLMODEL pca1 TYPE PRINCIPAL COMPONENT ANALYSIS ON ( SELECT x1, x2, x3 FROM mldemo.pca_poly ); Modified 0 rows ``` The model does not include `y` because PCA operates over only the input features. Examine the `machine_learning_models` and `principal_component_analysis_models` system catalog tables. ```sql SQL theme={null} SELECT importance FROM sys.machine_learning_models a, sys.principal_component_analysis_models b WHERE a.id = b.machine_learning_model_id AND name = 'pca1'; importance -------------------------------------------------------------------------------- [0.6022962127628679, 0.33286202821081917, 0.06484175902631285] Fetched 1 row ``` The `importance` value indicates that over 93 percent of the signal is in the first two PCA output features, which means it is possible to have a robust model even if you remove the third feature. This example uses the Polynomial Regression model to access the PCA output features. The example executes the `pca1` PCA function with the PCA input features followed by the number of features. This number starts at `1`, so this example uses PCA features `1` and `2`, but not the last feature. ```sql SQL theme={null} CREATE MLMODEL poly2 TYPE POLYNOMIAL REGRESSION ON ( SELECT pca1(x1, x2, x3, 1), pca1(x1, x2, x3, 2), y FROM mldemo.pca_poly ) options('order'->'2', 'metrics'->'true'); Modified 0 rows ``` Notice that the example still references all three input features, but this model now has only two independent variables. As a result, the model trains 15 percent faster than the version with three independent variables. It is also a much simpler model. The model has six terms in the polynomial regression instead of 10, when there were three independent variables. This query examines how well the model fits the data. ```sql SQL theme={null} SELECT coefficient_of_determination FROM sys.machine_learning_models a, sys.polynomial_regression_models b WHERE a.id = b.machine_learning_model_id AND name = 'poly2'; coefficient_of_determination ----------------------------- 0.9952839146818417 Fetched 1 row ``` The model is a great fit with more than 99 percent accuracy. In contrast, here is an example where PCA is not a good idea. This example goes back to the three-dimensional clusters data set used in the [Clustering Models](#clustering-models) examples. This example creates the PCA model and checks the `importance` value of the PCA output features in the `machine_learning_models` and `principal_component_analysis_models` system catalog tables. ```sql SQL theme={null} CREATE MLMODEL pca2 TYPE PRINCIPAL COMPONENT ANALYSIS ON ( SELECT x, y, z FROM mldemo.clusters_3d ); Modified 0 rows SELECT importance FROM sys.machine_learning_models a, sys.principal_component_analysis_models b WHERE a.id = b.machine_learning_model_id AND name = 'pca2'; importance -------------------------------------------------------------------------------- [0.3688781607553818, 0.3333390832624229, 0.2977827559821952] Fetched 1 row ``` In this case, all three PCA output features are fairly evenly weighted. By removing the last feature, the model covers only about 70% of the signal. This strongly indicates that proceeding with a two-feature model would be problematic and inaccurate. ### Linear Discriminant Analysis While PCA is unsupervised, linear discriminant analysis (LDA) is a supervised dimensionality reduction algorithm. LDA works only with numeric classification algorithms, but the classification can be binary or multi-class. Hence, LDA understands how the data should be used, making it operate differently. Creating an LDA model is mostly similar to PCA. One difference is that LDA also requires a class or label. This tutorial uses the three-dimensional cluster data that the PCA model struggled to work with. The example uses LDA to reduce the features of the cluster data by using the label already contained in the data set. ```sql SQL theme={null} CREATE MLMODEL lda1 TYPE LINEAR DISCRIMINANT ANALYSIS ON ( SELECT x, y, z, label FROM mldemo.clusters_3d ); Modified 0 rows ``` As with PCA, there is an `importance` column in the system catalog tables. ```sql SQL theme={null} SELECT importance FROM sys.machine_learning_models a, sys.linear_discriminant_analysis_models b WHERE a.id = b.machine_learning_model_id AND name = 'lda1'; importance -------------------------------------------------------------------------------- [1.0000000000000133, -1.4051695910079908E-20, -1.3342566035320652E-14] Fetched 1 row ``` This output strongly indicates that only the first LDA output feature matters, which means that it is possible to make a single feature classification model. This example takes the LDA-reduced model as input and uses it to create a neural network model, using the `FEEDFORWARD NETWORK` model. Note that this model uses the `cross_entropy_loss` option to enable multi-class classification. ```sql SQL theme={null} CREATE MLMODEL three_d_clusters_with_1_feature TYPE FEEDFORWARD NETWORK ON ( SELECT lda1(x, y, z, 1), CASE WHEN label = 'LEFT' THEN { { 1.0, 0.0, 0.0 } } WHEN label = 'RIGHT' THEN { { 0.0, 1.0, 0.0 } } ELSE { { 0.0, 0.0, 1.0 } } END AS target FROM mldemo.clusters_3d ) options( 'metrics'->'true', 'hiddenLayers'->'2', 'hiddenLayerSize'->'4', 'outputs'->'3', 'lossFunction'->'cross_entropy_loss', 'useSoftMax'->'true' ); Modified 0 rows ``` Execute the model for a new point to see if it returns a reasonable value. ```sql SQL theme={null} SELECT three_d_clusters_with_1_feature(lda1(0, 0, 0, 1)) AS predicted; predicted -------------------------------------------------------------------------------- [[0.007386760165583984, 0.0022085315302674473, 0.9904047083041486]] Fetched 1 row ``` In this case, the query asks for a prediction for the input `0, 0, 0`. This input first passes through the LDA model because that is what the neural network model was built on. The query asks the LDA model for its first component, and then runs the neural network on that value. The neural network outputs a vector of class probabilities; `VECTOR_ARGMAX` returns the 1‑based index of the highest probability. In this example, the highest probability class is the third class, which the query defines as `CENTER`. ```sql SQL theme={null} SELECT count(*) / 1000000.0 FROM mldemo.clusters_3d WHERE ( VECTOR_ARGMAX( three_d_clusters_with_1_feature(lda1(x, y, z, 1)) ) = 1 AND label = 'LEFT' ) OR ( VECTOR_ARGMAX( three_d_clusters_with_1_feature(lda 1(x, y, z, 1)) ) = 2 AND label = 'RIGHT' ) OR ( VECTOR_ARGMAX( three_d_clusters_with_1_feature(lda1(x, y, z, 1)) ) = 3 AND label = 'CENTER' ); (_count(*)_0)/((1000000.0)) ---------------------------- 0.818108 Fetched 1 row ``` This model is nearly 82 percent accurate despite being simplified from three features to one. While the accuracy is worse, the time required to train this model is greatly reduced. ## Related Links [Machine Learning Model Functions](/machine-learning-model-functions) [Clustering and Dimension Reduction Models](/clustering-and-dimension-reduction-models) # Clustering and Dimension Reduction Models Source: https://docs.ocient.com/clustering-and-dimension-reduction-models Train and apply OcientML clustering and dimensionality reduction models including k-means and PCA, with SQL syntax, model options, and example queries. supports these machine learning models for clustering and dimension reduction. To create the model, use the `CREATE MLMODEL` syntax. For details, see [CREATE MLMODEL](/machine-learning-model-functions). Model option names are case-sensitive. ## Principal Component Analysis Model Type: `PRINCIPAL COMPONENT ANALYSIS` Principal Component Analysis (PCA) is typically not used as a model on its own. PCA is most commonly used on the inputs to other models. PCA serves two purposes. 1. PCA normalizes all numeric feature data. Some types of models are sensitive to the scale of numeric features, and when different features have different scales, the results end up skewed. PCA normalizes all features to the same scale. 2. PCA is used for dimensionality reduction. PCA computes linear combinations of the original features to put the most signal into a smaller number of new features. The input result set when creating a PCA model is N numeric columns, which are all features. There is no label or dependent variable. After you create a PCA model, the `sys.principal_component_analysis_models` system catalog table contains information on the percentage of the signal in each PCA feature. You can use this information to figure out how many of the output features to keep. Also, after you create the model, you can see its details by querying the `sys.machine_learning_models` and `sys.machine_learning_model_options` system catalog tables. ### Principal Component Analysis and Logistic Regression You can also use PCA models as inputs to other models. For example, if you have three features and you want to use PCA to reduce the number to two features, you can execute the following SQL statements. ```sql SQL theme={null} CREATE MLMODEL reduceTo2 TYPE PRINCIPAL COMPONENT ANALYSIS ON ( SELECT c1, c2, c3 FROM public.my_table ); ``` You can use this model as input for another model, for example, logistic regression. ```sql SQL theme={null} CREATE MLMODEL binaryClass TYPE LOGISTIC REGRESSION ON ( SELECT reduceTo2(c1, c2, c3, 1), reduceTo2(c1, c2, c3, 2), label FROM … ); ``` To correctly use this model later, you must pass the original features through the PCA model when you execute the logistic regression model. ```sql SQL theme={null} SELECT binaryClass( reduceTo2(x1, x2, x3, 1), reduceTo2(x1, x2, x3, 2) ) FROM … ; ``` Similarly, to create a PCA analysis over four variables, execute this SQL statement: ```sql SQL theme={null} CREATE MLMODEL my_model TYPE PRINCIPAL COMPONENT ANALYSIS ON ( SELECT c1, c2, c3, c4 FROM public.my_table ); ``` ### **Model Options** `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ## K-Means Clustering Model Type: `KMEANS` K-means is an unsupervised clustering algorithm. All of the columns in the input result set are features, and there is no label. All the input columns must be numeric. The algorithm finds `k` points such that all points are classified by the closest `k` points. Distance calculations are Euclidean by default. ### **Model Options** #### Required `k` — This option must be a positive integer. The option specifies the algorithm for how many clusters to make. #### Optional `epsilon` — If you specify this option, the value must be a valid positive floating point value. When the maximum distance that a centroid moves from one iteration of the algorithm to the next is less than this value, the algorithm terminates. This parameter defaults to `1e-4`. `normalize` — If you set this option to `true`, the model normalizes the data before the start of training. The default value is `true`. `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ### **Execute the Model** Create a K-means model with `8` clusters. ```sql SQL theme={null} CREATE MLMODEL my_model TYPE KMEANS ON ( SELECT x1, x2, x3, x4 FROM public.my_table ) options( 'k' -> '8' ); ``` Also, after you create the model, you can see its details by querying the `sys.machine_learning_models` and `sys.machine_learning_model_options` system catalog tables. Because there are no labels for clusters, when you execute this function after training with the same number (and same order) of features as input, the result is an integer that specifies the cluster to which the point belongs. ```sql SQL theme={null} SELECT my_model(x1, x2, x3, x4) FROM my_table; ``` After you execute a model, you can find the details about the execution results in the `sys.k_means_models` system catalog table. For details, see the description of the associated system catalog tables in the Machine Learning section in the [System Catalog](/system-catalog#page-title). ## Gaussian Mixture Model Type: `GAUSSIAN MIXTURE MODEL` Gaussian Mixture Models (GMMs) find the mixture of `N` Gaussian distributions that best fit the data. The most common use is unsupervised clustering, but there are some key differences compared to K-means clustering: * GMMs can handle clusters that are not circular, i.e., clusters in the shape of ovals or higher-dimensional analogs with variances in different directions. * Clusters can have an arbitrary rotation, i.e., they can have covariance. * GMMs can handle instances when clusters are not as equally likely. If a point is right between two clusters, it is more likely to be more common in the training data. * GMMs can show the probability of a new point belonging to each cluster instead of providing only a single cluster value. GMMs generally handle more complex data than K-means models, but they require more training and processing time during execution. ### Model Options #### Required `numDistributions` — Must be a positive integer. This value specifies the number of clusters of Gaussian distributions for the model to make. #### Optional `epsilon` — If you specify this option, the value must be a valid positive floating point value. When the maximum distance that the entire best model moves in its n-dimensional space is less than this value, the algorithm terminates. This parameter defaults to 1e-8 (0.00000001 as a decimal number). `normalize` — If you set this option to `true`, the model automatically computes the mean and standard deviation of each feature and uses them to normalize the data during training. Defaults to `true`. `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ### Execute the Model Create a GMM with three clusters of Gaussian distributions. ```sql SQL theme={null} CREATE MLMODEL gmm TYPE GAUSSIAN MIXTURE MODEL ON ( SELECT x, y, z FROM my_table ) options( 'numDistributions' -> '3' ); ``` Also, after you create the model, you can see its details by querying the `sys.machine_learning_models` and `sys.machine_learning_model_options` system catalog tables. When you execute this function after training with the same number (and same order) of features as input, the result is an integer that specifies the probable cluster to which the point belongs. ```sql SQL theme={null} SELECT gmm(x,y,z) FROM my_table; ``` After you execute a model, you can find the details about the execution results in the `sys.gaussian_mixture_models` system catalog table. For details, see the description of the associated system catalog tables in the Machine Learning section in the [System Catalog](/system-catalog). ## Linear Discriminant Analysis Model Type: `LINEAR DISCRIMINANT ANALYSIS` Linear Discriminant Analysis (LDA) is a dimension-reduction technique similar to PCA. While PCA is unsupervised, LDA can classify and use class labels to find linear combinations of features that best separate the classes. The input result set when you create an LDA model is `N` numeric columns that are all features. The last column is a class label and can be any data type. After you create an LDA model, the `sys.linear_discriminant_analysis_models` system catalog table contains information on the percentage of the signal that is in each LDA feature. You can use this information to figure out how many of the output features to keep. Also, after you create the model, you can see its details by querying the `sys.machine_learning_models` and `sys.machine_learning_model_options` system catalog tables. ### Linear Discriminant Analysis and Logistic Regression You can use LDA models as inputs to other models. For example, to reduce the number of features from three to two using LDA, you can execute these SQL statements. ```sql SQL theme={null} CREATE MLMODEL reduceTo2 TYPE LINEAR DISCRIMINANT ANALYSIS ON ( SELECT c1, c2, c3, label FROM public.my_table ); ``` You can use this model as input for another model, for example, logistic regression. ```sql SQL theme={null} CREATE MLMODEL binaryClass TYPE LOGISTIC REGRESSION ON ( SELECT reduceTo2(c1, c2, c3, 1), reduceTo2(c1, c2, c3, 2), label FROM … ); ``` To correctly use this model later, you must pass the original features through the LDA model when you execute the logistic regression model. ```sql SQL theme={null} SELECT binaryClass( reduceTo2(x1, x2, x3, 1), reduceTo2(x1, x2, x3, 2) ) FROM … ; ``` Similarly, to create an LDA model over four variables, execute this SQL statement. ```sql SQL theme={null} CREATE MLMODEL my_model TYPE LINEAR DISCRIMINANT ANALYSIS ON ( SELECT c1, c2, c3, c4, label FROM public.my_table ); ``` ### **Model Options** `normalize` — If you set this option to `true`, the model automatically computes the mean and standard deviation of each feature and uses them to normalize the data during training. Defaults to `true`. `featureArray` — If you set this option to `true`, the model expects only one array-type column as input instead of multiple columns of training data. Each array row in the input column must be of the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. ### Execute the Model When you execute the trained LDA model, you must provide the same original input features in the same order, followed by a positive integer argument that specifies the LDA component to return. The LDA component index starts at 1. ```sql SQL theme={null} SELECT my_model(col1, col2, col3, col4, 2) as component2, my_model(col1, col2, col3, col4, 3) as component3, FROM public.my_table; ``` ## Bibliography Bahmani, Bahman, Benjamin Moseley, Andrea Vattani, Ravi Kumar, and Sergei Vassilvitskii. “Scalable K-Means++.” Proceedings of the VLDB Endowment 5, no. 7 (2012): 622–33. [https://doi.org/10.14778/2180912.2180915](https://doi.org/10.14778/2180912.2180915). ## Related Links [Classification Models](/classification-models) [Other Models](/other-models) [Machine Learning Models](/machine-learning-models) # # Commands Supported by the Ocient JDBC CLI Program Source: https://docs.ocient.com/commands-supported-by-the-ocient-jdbc-cli-program Reference for commands supported by the Ocient JDBC CLI program, including session management, query execution, file output, and SQL script execution. Commands are not case-sensitive in the JDBC CLI program. For DDL statements, see [Data Definition Language (DDL) Statement Reference](/data-definition-language-ddl-statement-reference). All commands, except QUIT, must end with a semicolon at the command line. You can cancel any query that does not update data by pressing CTRL+C. The Ocient JDBC driver supports multi-statement transactions. By default, a connection uses the autocommit mode, in which the database commits each statement immediately. To group several statements into a single transaction, disable autocommit with [SET AUTOCOMMIT](#set-autocommit), and then end the transaction with [COMMIT](#commit) or [ROLLBACK](#rollback). The HTTP Query API does not support transactions. To use transactions, connect to the Ocient System with the JDBC driver or the pyocient module. The Connector supports transactions and performs the commit and rollback actions automatically. When you use the Spark Connector, you do not need to issue COMMIT or ROLLBACK commands. ### CANCEL Cancels the current query that is still running on the system. **Syntax** ```shell Shell theme={null} CANCEL uuid; ``` | **Parameter** | **Data Type** | **Description** | | ------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | uuid | String | A query Universally Unique IDentifier (UUID) that is currently running. Use [LIST ALL QUERIES](#list-all-queries), or search the `sys.queries` table to find the UUID. You must enclose the UUID in single quotes, such as `'c87b506f-66d6-4987-8bf7-1fd4a06b64b1'`. | ### COMMIT Commits the current transaction and permanently saves all changes made after the transaction started. This command applies only when you disable autocommit with [SET AUTOCOMMIT](#set-autocommit). When autocommit is enabled, COMMIT has no effect. **Syntax** ```shell Shell theme={null} COMMIT [ TRANSACTION ]; ``` Entering the COMMIT command with no other arguments commits the current transaction on the connection. This action is equivalent to calling the `commit()` method on the JDBC Connection object. ### CONNECT Connects to a SQL Node. For details on the connection steps, see [Connect Using JDBC](/connect-using-jdbc). **Syntax** ```shell Shell theme={null} CONNECT TO [ USER username USING password ] ``` The `` parameter must follow this format: `jdbc:ocient://node_address1:port1[,node_address2:port2[,...]]/database_name[;property=value[;...]]` If you do not specify the `username` and `password`, then the database uses the default credentials specified at startup. **JDBC URL parameters** | **Parameter** | **Description** | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `node_address` | A hostname or IP address.
The database attempts to connect using each `DNSName:Port` pair in the comma-delimited list from left to right until a successful connection happens. | | `port` | Standard JDBC port number is `4050`.
For details about ports, see [Ports](/ocient-simulator#ports). | | `database_name` | The identifier used for your database. | **Additional JDBC URL Parameters** The JDBC URL can include these optional parameters in the key-value format `property=value`. Separate each parameter pair by semicolons. URL parameters are case-insensitive. To authenticate using Single Sign-On, set the `handshake` parameter to `SSO` and leave the `user` and `password` parameters empty. For example: `jdbc:ocient://DNSName:Port[,IP2:Port2,...]/databaseName;handshake=SSO;user=;password=;` | **Parameter** | **Description** | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bulkLoadChunkSize` | The number of rows to include in each JSON data file (chunk) uploaded. The default value is `60000`. | | `bulkLoadCleanupOnFailure` | If you set this parameter value to `true`, the driver deletes the temporary files and pipelines even if the load fails. The default value is `true`. | | `bulkLoadFailOnError` | If you set this parameter value to `true`, `executeBatch()` fails if any `INSERT` operation fails.

If you set this parameter value to `false`, `executeBatch()` falls back to a standard multi-row `INSERT` operation if any INSERT operation fails. The default value is `false`. | | `bulkLoadLoaderNodesCacheSeconds` | The number of seconds to cache the list of active Loader Nodes the system discovers during bulk load. When you set this parameter to a value greater than `0`, the driver caches the Loader Node list for the specified duration and forwards the same value as a server-side `CACHE_MAX_TIME` hint on the lookup query. This cache is -wide and is not specific to individual connections. Do not enable this parameter if your application connects to multiple Ocient clusters from the same JVM. The default value is `0` (disabled). | | `bulkLoadMode` | The bulk load transport mode. Set this parameter to `ssh` to stage data using SSH/SFTP to Loader Nodes, or set it to `s3` to stage data in an S3-compatible object store. The default value is `ssh`. | | `bulkLoadPollIntervalSeconds` | The number of seconds to wait between polling the `sys.pipelines` system catalog table for load status. The default value is `2`. | | `bulkLoadS3AccessKeyId` | The access key identifier for authenticating to the S3 endpoint. This parameter is required when you set the `bulkLoadMode` parameter to `s3`. | | `bulkLoadS3ApiCallTimeoutMs` | The timeout, in milliseconds, for individual S3 API calls. The default value is `120000` (two minutes). | | `bulkLoadS3Bucket` | The S3 bucket name for staging bulk load data files. This parameter is required when you set the `bulkLoadMode` parameter to `s3`. | | `bulkLoadS3EnablePathStyleAccess` | If you set this parameter to `true`, the driver uses path-style access for S3 requests (e.g., `http://endpoint/bucket/key` instead of `http://bucket.endpoint/key`). This parameter is required for S3-compatible services such as MinIO or Ceph. The default value is `true`. | | `bulkLoadS3Endpoint` | The S3-compatible endpoint URL for bulk load staging (e.g., `https://s3.us-east-1.amazonaws.com`). This parameter is required when you set the `bulkLoadMode` parameter to `s3`. | | `bulkLoadS3MultipartPartSize` | The size, in bytes, of each part in a multipart upload. The minimum value is `5242880` (5 MiB). The default value is `10485760` (10 MiB). | | `bulkLoadS3MultipartThreshold` | The size, in bytes, above which the driver uses multipart upload instead of a single PUT request. The default value is `10485760` (10 MiB). | | `bulkLoadS3Prefix` | The key prefix for staged objects in S3. The default value is `ocient-bulk-load/`. | | `bulkLoadS3Region` | The AWS region for the S3 endpoint. The default value is `us-east-1`. | | `bulkLoadS3SecretAccessKey` | The secret access key for authenticating to the S3 endpoint. This parameter is required when you set the `bulkLoadMode` parameter to `s3`. | | `bulkLoadS3UploadConcurrency` | The number of data chunks to upload to S3 in parallel. Increasing this value can improve staging throughput for large batches. The default value is `1`. | | `bulkLoadSshChannelTimeoutSeconds` | The number of seconds to wait for an SFTP channel to open on an established SSH connection during bulk load. If the channel does not open within this duration, the driver throws an error. The default value is `30`. | | `bulkLoadSshConnectTimeoutSeconds` | The number of seconds to wait for the SSH connection and authentication to a Loader Node during bulk load. If the system does not establish a connection within this duration, the driver throws an error. The default value is `10`. | | `bulkLoadSshKeyPath` | The absolute path to the password-less private SSH key file. The default value is `~/.ssh/id_rsa`. | | `bulkLoadSshHostKeyVerification` | Controls how the JDBC driver verifies an SSH server host key for bulk load connections.

Supported values are:

`acceptAll` (default) — The driver accepts all host keys. The connector does not check the server host key against the `known_hosts` file and does not output warning logs.
`strict` — The driver accepts only hosts with keys present in the `known_hosts` file (see the `bulkLoadSshKnownHostsPath` parameter). The driver rejects unknown hosts or hosts with changed keys.
`acceptNew` — The driver accepts new hosts and adds their keys to the `known_hosts` file, but rejects hosts with keys changed from the data in `known_hosts`. | | `bulkLoadSshKnownHostsPath` | The path to the `known_hosts` file, which contains SSH verification keys. The system uses this path only if the `bulkLoadSshHostKeyVerification` parameter is set to `strict` or `acceptNew`.

The default path is `~/.ssh/known_hosts`. | | `bulkLoadSshUser` | The SSH username to use when connecting to Loader Nodes. | | `bulkLoadThreshold` | The minimum number of rows in a batch group to trigger a bulk load. The default value is `25000`. | | `bulkLoadUseAtomicPipeline` | If you set this parameter to `true`, bulk load uses an atomic pipeline that creates, starts, and monitors the pipeline in a single blocking operation `(CREATE TRANSACTIONAL PIPELINE ... START FOREGROUND)`. This mode requires the Ocient System version 27.1 or later.

Set this parameter to `false` to use the legacy create, start, monitor, and drop functionality when connecting to older servers. The default value is `true`. | | `defaultSchema` | Default schema. | | `enableBulkLoad` | Enables the high-speed bulk load feature. Set this parameter value to `true` to enable this feature. Otherwise, set this parameter to `false` to leave the feature disabled. The default value is `false`. | | `force` | If set to true (case-sensitive), this parameter disables load-balancing for the connection. | | `handshake` | Specifies the handshake protocol used for the connection.
Supported options include: `"CBC", "GCM", "SSO"`
`"GCM"` — (Galois/Counter Mode). This is the default encryption and is the recommended password encryption algorithm.
`"CBC"` — (Cipher Block Chaining) for password encryption.
`"SSO"` — Single Sign-On. | | `identityprovider` | An SSO integration established in the database. For details, see [CREATE SSO INTEGRATION](/cluster-and-node-management#create-sso-integration). | | `logfile` | The filename to use for JDBC tracing. | | `loglevel` | If set to ERROR (warnings and errors only) or DEBUG (verbose tracing) and logfile is also set, JDBC tracing is enabled. This parameter is case-sensitive. | | `longQueryThreshold` | Estimated query runtime in milliseconds before deeper query optimization runs.
`0` — Use database server default.
`-1` — Never run deeper optimization. | | `maxRows` | Maximum allowed result set size in the number of rows. | | `maxRowsPerInsertBatch` | The maximum number of rows to combine into a single multi-row `INSERT` SQL statement when executing a batch load. If the total number of rows in a batch group exceeds this limit, the driver splits the group into multiple sub-statements.

The default value is 128,000 rows. | | `maxTempDisk` | Maximum allowed temp disk usage as a percentage (0 - 100). | | `maxThreadsPerResultSet` | The maximum number of threads the client uses to fetch rows from the server per Result Set, as defined in the [official JDBC documentation](https://docs.oracle.com/javase/7/docs/api/java/sql/ResultSet.html). In this context, a thread represents a logical connection to the database.
Applications that create and operate on Statement objects concurrently might find value in setting this parameter. Defaults to `0`, which effectively creates an unbounded Result Set thread pool. | | `maxTime` | Maximum allowed runtime of a query in seconds before it is canceled on the server. | | `networkTimeout` | Network connection timeout in milliseconds.
If unspecified, this defaults to 10000 milliseconds. | | `parallelism` | Limits a query to running on a specified number of cores on each CPU. | | `password` | The password for the user. | | `priority` | Query priority. This sets the priority for queries to run on the server side.
If unspecified, this defaults to `1.0`. | | `priorityAdjustFactor` | The default query priority adjustment value. For details, see [SET ADJUSTFACTOR](/commands-supported-by-the-ocient-jdbc-cli-program#set-adjustfactor). | | `priorityAdjustTime` | The default frequency to adjust the query priority. For details, see [SET ADJUSTTIME](/commands-supported-by-the-ocient-jdbc-cli-program#set-adjusttime). | | `serviceClassName` | Specifies the name of the service class to use for the database session. | | `sparkMode` | When you set this parameter to `true`, the driver enables -specific JDBC behavior intended to improve compatibility with the Ocient Spark connector. For details, see [JDBC Spark Connector](/jdbc-spark-connector).

This parameter defaults to `false` if you are connecting directly using the Ocient JDBC driver (e.g., connecting with the `DriverManager` class or CLI).
Otherwise, if you are using the Spark connector (catalog or `.format("ocient")`), the default is `true`. | | `ssoNumericAddress` | Specifies the SSO callback URL as `127.0.0.1`. When this value is `false`, the URL is `localhost`. The default value is `false`. | | `ssoOAuthCodeCallbackPort` | If `ssoOAuthFlow=authorizationCode`, this parameter specifies the port that the SSO authorization uses.
If unspecified, the default value is `7050`. | | `ssoOAuthFlow` | This parameter is only applicable if SSO authorization is enabled.
Forces the driver to use either the "authorizationCode" or "deviceGrant" flow to establish a Single Sign-On session.
If this parameter is not provided, the System uses the "authorizationCode" flow when a web browser is available to the client and the "deviceGrant" flow when a web browser is not available to the client. | | `ssoDebugMode` | When set to `true`, the system records additional log messages related to SSO. | | `ssoOktaNativeTokenPath` | The path to the Native Token (AES-256GCM JWE). The path is relative to your home directory (i.e. '\~/'). | | `ssoSslCallback` | Specifies whether to use HTTPS instead of HTTP for the SSO callback URL. The default value is `false`. | | `ssoTimeoutSeconds` | The number of seconds before the SSO connection request times out. The default value is `60`. | | `statementPooling` | ℹ️ This parameter has been removed as of Ocient JDBC version 2.104 and later.
When set to `ON`, recently used statements are cached.
Set to `OFF` when using a third-party application that implements statement pooling | | `tls` | Can be set to `unverified` or `on`. Enables SSL/TLS encryption for the connection. | | `timeoutMillis` | Number of milliseconds before cancellable operations are timed out and killed by the driver. 0 means no timeout. Default: 0 | | `user` | The identifier of the user. | The JDBC driver also supports system properties that apply globally across all connections. To set these properties, use the `-D` flag at JVM startup. For details, see [JVM System Properties](/jdbc-manual#jvm-system-properties). ### DESCRIBE TABLE Lists the columns of the specified table and its associated data types. **Syntax** ```shell Shell theme={null} DESCRIBE TABLE [schema.]table [verbose] ``` | **Parameter** | **Description** | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `table` | A table identifier. | | `schema` | A qualifying schema name for the specified table.
If a qualifying schema is not specified for the table, the default name schema is assumed. | | `verbose` | If specified, the Ocient System prints the full column metadata for the specified table as returned by GetColumns. | ### DESCRIBE VIEW Returns the query text used to create the specified view. **Syntax** ```shell Shell theme={null} DESCRIBE VIEW [schema.]view [verbose] ``` | **Parameter** | **Description** | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `view` | A view identifier. | | `schema` | A qualifying schema name for the specified table.
If a qualifying schema is not specified for the table, the default name schema is assumed. | | `verbose` | If specified, the Ocient System prints the full column metadata for the specified table as returned by GetColumns. | ### EXTRACT Extracts a result set in delimited files to a specified location. For details about using this command, see [Data Extract Tool](/data-extract-tool). **Syntax** ```sql SQL theme={null} EXTRACT TO location { LOCAL | S3 } [ OPTIONS ( [ param=value [ ,... ] ] ) ] AS query ``` ### GET JDBC VERSION Returns the JDBC version of the driver. **Syntax** ```shell Shell theme={null} GET JDBC VERSION ``` ### GET SCHEMA Retrieves the default name schema for tables. **Syntax** ```shell Shell theme={null} GET SCHEMA ``` ### KILL Issues a hard kill command to the virtual machine to terminate a query running on the system. Use only if `CANCEL` fails. **Syntax** ```shell Shell theme={null} KILL uuid ``` | **Parameter** | **Description** | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `uuid` | A query UUID that is currently running. Use [LIST ALL QUERIES](#list-all-queries), or search the `sys.queries` table to find the UUID. | ### LIST ALL QUERIES List all the currently executing queries on the database. This is the equivalent of executing `SELECT * FROM sys.queries`. **Syntax** ```shell Shell theme={null} LIST ALL QUERIES ``` ### LIST INDEXES Lists the indexes on the specified table and its columns. **Syntax** ```shell Shell theme={null} LIST INDEXES [schema.]table [verbose] ``` | **Parameter** | **Description** | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `table` | A table identifier. | | `schema` | A qualifying schema name for the specified table.
If a qualifying schema is not specified for the table, the default name schema is assumed. | | `verbose` | If specified, the Ocient System prints the full column metadata for the specified table as returned by getIndexInfo. | ### LIST SYSTEM TABLES Lists all system catalog tables in the database. **Syntax** ```shell Shell theme={null} LIST SYSTEM TABLES [verbose] ``` | **Parameter** | **Description** | | ------------- | --------------------------------------------------------------------------------------- | | `verbose` | If verbose is specified, the full table metadata, as returned by getTables, is printed. | ### LIST TABLES Lists all tables (in all schemata) in the database. **Syntax** ```shell Shell theme={null} LIST TABLES [verbose] ``` | **Parameter** | **Description** | | ------------- | --------------------------------------------------------------------------------------- | | `verbose` | If verbose is specified, the full table metadata, as returned by getTables, is printed. | ### LIST VIEWS Lists all views (in all schemata) in the database. **Syntax** ```shell Shell theme={null} LIST VIEWS [verbose] ``` | **Parameter** | **Description** | | ------------- | -------------------------------------------------------------------------------------- | | `verbose` | If verbose is specified, the full table metadata, as returned by getViews, is printed. | ### OUTPUT GIS KML Outputs the next query in KML format to the specified filename. This flag affects only the next query. This syntax prints out all GIS types to the KML file, and non-GIS types are added to the description of elements in the same row. You can upload this KML file to certain visualization tools. **Syntax** ```shell Shell theme={null} OUTPUT GIS KML ``` | **Parameter** | **Description** | | ------------- | ------------------------------ | | `filename` | A filename for the KML output. | ### OUTPUT NEXT QUERY Output the next query in CSV format to the specified filename. This flag only affects the next query. All subsequent queries print out their results. Add `APPEND` to add output to the specified file instead of overwriting it. **Syntax** ```shell Shell theme={null} OUTPUT NEXT QUERY filename [APPEND] ``` | **Parameter** | **Description** | | ------------- | ------------------------------ | | `filename` | A filename for the CSV output. | ### PERFORMANCE Enables different levels of query result output for measuring benchmarks by using different performance options. **Syntax** ```shell Shell theme={null} PERFORMANCE [ NETWORK | DATABASE | OFF ] ``` | **Parameter** | **Description** | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OFF`
| This is the default mode.
The system executes a query through all normal stages, including the database, client network, and client processing. | | `NETWORK` | The system executes a query through the database and client network, but discards rows before they are processed by the client (nothing is printed to stdout).
Use this mode when you think the client network is hindering performance. | | `DATABASE` | The system executes a query through the database only. The database does not send any matching rows to the client.
Use this mode when you think the database query execution is affecting performance. | `PERFORMANCE ON` mode has been removed as of JDBC version 3.0. ### QUIT Closes the command-line interface. **Syntax** ```shell Shell theme={null} QUIT ``` ### REPORT LOG ON Enables -based metrics collection and redirects reporting events to the configured appenders. The `REPORT LOG ON` command requires JDBC driver version 2.88 or later. The Log4j configuration determines the destination for report logs. The Ocient logger name is `com.ocient.util.ReportLogger`. For details about configuring Log4j, see [the Log4j documentation](https://logging.apache.org/log4j/2.x/manual/configuration.html). The default Log4j configuration routes report logs to the `RollingFileAppender`. The default configuration includes these settings: **Properties** * `fileName="logs/${sys:ocient.jdbc.sessionId:-unknown}/report.log"` * `filePattern="logs/${sys:ocient.jdbc.sessionId:-unknown}/%d{yyyy-MM-dd}-%i-report.log.gz"` **Rollover Policies** * `TimeBasedTriggeringPolicy` * `SizeBasedTriggeringPolicy (100MB)` You can retrieve the default configuration from the `ocient-jdbc4` jar file by using an archive utility like **unzip**. For example: ```shell Shell theme={null} unzip -p log4j2.xml ``` **Syntax** ```shell Shell theme={null} REPORT LOG ON [ interval ] ``` | **Parameter** | **Description** | | ------------- | ----------------------------------------------------------------------------------------------- | | `interval` | Optional. Sets the Log4j reporting interval in seconds.
If unspecified, defaults to `30`. | ### REPORT LOG OFF Disables file-based metrics collection. The `REPORT LOG OFF` command requires JDBC driver version 2.88 or later. **Syntax** ```shell Shell theme={null} REPORT LOG OFF ``` ### ROLLBACK Rolls back the current transaction and discards all changes made after the transaction started. This command applies only when you disable autocommit with [SET AUTOCOMMIT](#set-autocommit). When autocommit is enabled, ROLLBACK has no effect. **Syntax** ```shell Shell theme={null} ROLLBACK [ TRANSACTION ]; ``` Entering ROLLBACK with no other arguments rolls back the current transaction on the connection. This action is equivalent to calling the `rollback()` method on the JDBC Connection object. ### SELECT Retrieves the appropriate result set of a SQL `SELECT` statement. **Syntax** ```shell Shell theme={null} SQL select statement ``` ### SET AUTOCOMMIT Enables or disables the autocommit mode for the connection. When autocommit is enabled, the database commits each statement immediately. This mode is the default. When autocommit is disabled, the database groups statements into a transaction that you complete with the [COMMIT](#commit) or [ROLLBACK](#rollback) commands. When you change autocommit from `OFF` to `ON`, the database commits any pending transactions. **Syntax** ```shell Shell theme={null} SET AUTOCOMMIT { ON | OFF } ``` Enables or disables autocommit mode with values `ON` or `OFF`, respectively. When you enter the `ON` value, the database commits each statement immediately. This value is the default. When you enter the `OFF` value, the database groups subsequent statements into a transaction until you enter the COMMIT or ROLLBACK command. ### SET MAXROWS Sets the maximum number of rows allowed in a result set. Applies only to the statement object that executed the command. **Syntax** ```shell Shell theme={null} SET MAXROWS num_of_rows ``` | **Parameter** | **Description** | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `num_of_rows` | The maximum number of rows allowed in a result set. If a query returns more rows than the specified value, the result set is truncated by `num_of_rows`.
To disable this setting, you can either set the maximum rows to 0 or use the `RESET` keyword (e.g., `SET MAXROWS RESET;`). | ### SET ADJUSTFACTOR Adjusts the query priority by a specified percentage amount at every interval set by the [SET ADJUSTTIME](#set-adjusttime) command. The `SET ADJUSTFACTOR` command requires JDBC driver version 2.52 or later. **Syntax** ```shell Shell theme={null} SET ADJUSTFACTOR priorityAdjustFactor ``` | **Parameter** | **Description** | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `priorityAdjustFactor` | A percentage amount to raise or lower the query priority. Accepted values are in the range of \[-1,1].
At every interval set by the `priorityAdjTime`, the query priority is multiplied by `1 + priorityAdjustFactor` to get a new priority value. | ### SET ADJUSTTIME Modifies how frequently the query priority is adjusted during the course of execution. The `SET ADJUSTTIME` command requires JDBC driver version 2.52 or later. **Syntax** ```shell Shell theme={null} SET ADJUSTTIME priorityAdjTime ``` | **Parameter** | **Description** | | ----------------- | ---------------------------------------------------------- | | `priorityAdjTime` | The frequency for adjusting the query priority in seconds. | ### SET MAXTEMPDISK Sets the maximum percentage of temporary disk used by queries. Applies only to the statement object that executed the command. **Syntax** ```shell Shell theme={null} SET MAXTEMPDISK percentage ``` | **Parameter** | **Description** | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `percentage` | The maximum percentage of temporary disk used by queries. Any queries exceeding this threshold are killed.
To disable this setting, use the `RESET` keyword (e.g., `SET MAXTEMPDISK RESET;`). | ### SET MAXTIME Sets the maximum query time. Applies only to the statement object that executed the command. **Syntax** ```shell Shell theme={null} SET MAXTIME seconds ``` | **Parameter** | **Description** | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `seconds` | The maximum query time. Queries longer than this are killed.
To disable this setting, you can either set the maximum time to 0 or use the `RESET` keyword (e.g., `SET MAXTIME RESET;`). | ### SET PARALLELISM Sets a limit on the number of cores on each CPU that can be allocated to running the query. Applies only to the statement object that executed the command. The `parallelism` value limits a query to running on the specified number of cores on each CPU. **Syntax** ```shell Shell theme={null} SET PARALLELISM parallelism ``` | **Parameter** | **Description** | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `parallelism` | Limits a query to running on a specified number of cores on each CPU.
To disable this setting, you can either set the `parallelism` value to -1 or use the `RESET` keyword (e.g., `SET PARALLELISM RESET;`). | ### SET PRIORITY Sets the scheduling priority to use for queries. Applies only to the statement object that executed the command. **Syntax** ```shell Shell theme={null} SET PRIORITY priority ``` | **Parameter** | **Description** | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `priority` | The scheduling priority to use for queries.
Set from 0 to 100. To disable this setting, you can either set the priority to -1 or use the `RESET` keyword. (e.g., `SET PRIORITY RESET;`). | ### SET SCHEMA Changes the default schema for tables. Applies only to the statement object that executed the command. **Syntax** ```shell Shell theme={null} SET SCHEMA schema ``` | **Parameter** | **Description** | | ------------- | ---------------------------------- | | `schema` | The default schema for new tables. | ### SET SERVICECLASS Limits a query to using the specified service class. Applies only to the statement object that executed the command. **Syntax** ```shell Shell theme={null} SET SERVICECLASS service_class_name ``` | **Parameter** | **Description** | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `service_class_name` | The service class that a query is limited to using.
To disable this setting, you can use the `RESET` keyword. (e.g., `SET SERVICECLASS RESET;`). | ### SOURCE Reads and executes commands found in the specified file. **Syntax** ```shell Shell theme={null} SOURCE filename ``` | **Parameter** | **Description** | | ------------- | ------------------------- | | `filename` | A valid filename or path. | ### TIMING Enables or disables reporting the execution time of each query. **Syntax** ```shell Shell theme={null} TIMING { ON | OFF } ``` ## Related Links [JDBC Manual](/jdbc-manual) [System Catalog](/system-catalog) # Compute-Adjacent Input and Output on Large Working Sets Source: https://docs.ocient.com/compute-adjacent-input-and-output-on-large-working-sets Ocient compute-adjacent storage architecture co-locates compute and NVMe storage to deliver high throughput on large working sets and analytic queries. The storage of the data warehouse and the input and output (I/O) layer is built using a (CASA). When contrasted against modern cloud-based database engines, this means that the majority of the data to be analyzed is stored on NVMe SSDs directly attached to a significant fraction of the computational capacity of the system using the local PCIe busses of the nodes that make up the ***foundation level*** of the database engine. This approach, which is similar to a traditional MPP architecture, is designed to capitalize on the performance characteristics of NVMe SSDs when executing queries against a large ***working set*** of data. It is important to note that when executing a query, a specific record could be included as part of the results, but I/O and computational effort must be undertaken to make that determination. A working set, therefore, is defined as the set of records stored in the database that are considered as part of query execution, even if those records are eventually filtered out or not returned. A "large" working set is one that is substantially larger, even by multiple orders of magnitude, than the available DRAM on the set of nodes performing the computations. When this is the case, ***remote-object*** database engines that store the majority of their data in a separate storage layer are substantially affected by their inability to keep all the potentially-addressed records near and accessible to computation at the same time. Depending on the query load and their exact engine design and scheduling, query execution times negatively suffer from both simple bottlenecks due to network link capacity and "cache thrashing" that produces expensive extra drive I/O. Large working sets can be realized in two major ways: 1. Queries that regularly consider a substantial fraction of the entire data set. For example, regular querying or analysis of all data spanning years can mean a single query addresses most of the records in the database. 2. A large number of smaller queries that all consider a disjoint subset of the whole data set. When these queries are executed in parallel the sum total of the disjoint sets of addressed records can also approach most of the records in the database. The database engine in the System is a world-class relational database engine that can go toe-to-toe with other modern engines on small working sets but won’t necessarily out-perform them. After all, well known algorithms like sorting and hashing can only be made so fast when everything is in memory. But the performance distinctions of Ocient derive from its ability to scale CASA to keep all of the data close to the computation. ## NVMe SSDs in CASA One of the biggest drivers for the development of the was the ongoing shift from spinning HDDs to NVMe SSDs for data storage. SSD prices continue to fall, relative to HDDs, and there remains little doubt that they will eventually supplant HDDs for all but esoteric or extreme bulk storage applications. To ride this storage wave, the Ocient engine has been built to minimize the cost of using these drives as the primary storage tier of the engine while maximizing their performance. There are two major aspects to cost management: 1. *Minimization of the total number of stored bytes needed to achieve reliable access to data in a performant manner*. This is achieved using a novel and tunable ***erasure coding*** scheme in conjunction with a variety of compression techniques. This is detailed more thoroughly in the following sections. 2. *Consideration for endurance management in I/O operations against the NVMe SSDs.* The Ocient storage and execution engine is designed to minimize the number of I/O write operations undertaken against the storage media in order to maximize its lifetime and reduce drive replacement rate due to endurance wear. At a high level, this usually means minimizing small or random writes to the drives and instead favoring larger and sequential ones. To achieve this, the storage layer uses a specially-written and database-aware file system underneath storage structures designed to capitalize on the large volume of low-mutability data being stored. Using these mechanisms, the storage layer is able to collate writes in a manner that produces relatively large (from the drive’s point of view) ranges of contiguous blocks, ordered in the hundreds of KiB to hundreds of MiB in size. ## Erasure Coding Of particular note in the storage and I/O layer of the OcientAIQ Unified Data Platform is the novel use of a tunable erasure coding scheme for data storage. You can think of erasure coding as a generalization of simple XOR parity calculations. In a simple error correction scheme, you can apply the XOR operation on a sequence of bits: `B0, B1, B2, ..., Bn` such that: `B0 XOR B1 XOR ... XOR Bn = Bx`. An easily provable property of the XOR operation is that for any single Bi with 0 ≤ i ≤ n, that `B0 XOR ... XOR Bi-1 XOR Bi+1 XOR ... XOR Bn XOR Bx = Bi`. You can determine any single bit by performing the XOR operation on the other original bits and the extra bit together. In the context of error correction, this means that for the cost of an extra bit of storage (or memory) a system is able to detect, and even correct for, the loss of a single bit. There are a variety of error correction schemes described in computer science and found in the industry. In the application of this concept to database storage, the Ocient storage layer extends it from individual bits, and beyond bytes, to an error correction unit of the 4KiB block (which is notably aligned with the I/O resolution of standard NVMe SSD drives). It also further extends from enabling single loss or failure to a fully tunable approach that uses ***XOR***, ***P+Q***, or ***Cauchy-Reed-Solomon*** coding that allows the user to specify exactly how many faults: one, two, or K, respectively. Like the simple XOR formulation, these schemes utilize mathematical computations (in this case, polynomials over a finite/Galois field) to compute additional blocks of data that enable reconstructing a configurable number of missing blocks. In database and storage systems that utilize replication to provide data reliability and availability, it is common to describe storage in terms of the number of replicas (e.g., 3x replication allows for up to three failures). The obvious implication of replication is that each replica is a full copy of the original data and represents a multiple of the minimal data storage required. In an erasure coding scheme, the overhead for `K` tolerable faults is calculated as `(N + K)/N`, where `N` is the total width and `K` is parity width. If `N` is 10 (take that as a given for the moment) then to support a maximum of 2 faults (the equivalent of 3x replication, which includes the original data and two copies) the system needs only `12 / 10 = 1.2x` times the original storage size, contrasted with the 3x of the replication scheme. The utilization of space-efficient erasure coding can be found in a variety of storage and transmission systems, but, in a classic space-time trade-off, the space savings come at a computational cost. The approach taken in the Ocient storage layer is novel, at least for databases, and is designed to mitigate the computational overhead of erasure coding. If one considers a "file" of some size as a unit of storage, a basic approach for applying an erasure coding scheme to provide loss tolerance to this file would be to break it into `N` chunks and perform the requisite computation to produce an additional `K` chunks. At read time, the system would select an available subset of the `N+K` chunks (possibly preferring the original `N`) and perform the **rebuild** computation to produce the original file for access. In this framing the unit of fault tolerance is the individual file, and rebuilding computational overhead can be found in the read path. However, in a database designed for OLAP-style queries, the potential for the overhead of any rebuilding computation in the read path even when no drives or nodes are offline is troublesome. Additionally, when the system is in a degraded state, the potential need to rebuild an entire file only to retrieve a small subset of its data is expensive as well. The Ocient storage layer, by directly implementing the erasure coding scheme and integrating with it, avoids these issues in two major ways. The first advantage is the unit of storage. Ocient uses storage units named ***segments***, which are not chunked into `N` pieces. Instead, `N` separate and similarly sized segments are considered together as a unit named a ***segment group***. The erasure coding scheme is then applied across the segments to produce `K` additional "segments" worth of redundancy information, which, in a very loose application of the term, is named ***parity data***. This graphic shows a high-level illustration of how Ocient stores each data segment. Segments contain all the data needed for a subset of table rows, including the actual column data as well as indexes, metadata, and statistical data such as probability density functions (PDFs) and count distinct estimates (CDEs). In addition, the erasure coding scheme also includes parity data for other segments in the same segment group. Example segment and segment part structure that contains data, statistics, and index parts and corresponding parity data The newly generated parity data is then broken up and evenly distributed amongst the original segments (in a particular and calculated way) such that each segment is now approximately `(N+K)/N` times larger than it originally was, and any missing segment from a particular segment group can be reconstructed when one has access to any set of `N` segments from the same group. The Ocient storage layer ensures that each segment from a group is stored on a separate physical node, thereby providing data availability in the event of up to `K` node losses. Another important feature of this scheme is due to the fact that a given record is always stored in a single segment. Because the segments themselves are not chunked, any specified record is guaranteed to exist intact on some node/drive, and readable with no overhead incurred when that drive is online. Put another way, in the non-degraded state, the reliability scheme implemented incurs no computational cost. The second advantage of the Ocient System is each segment acts as a self-contained unit of information, containing not only database records but also metadata and indexes on those records. These sub-units within the segment, named ***segment parts*** are individually addressable and independently rebuildable with 4KiB resolution. The implication of this is that in a degraded state (where a node or drive is missing), it is not strictly necessary to rebuild all missing segments to service a specified query. Instead, the storage layer is capable of addressing and rebuilding only the parts and even single blocks of parts required for it. This often greatly reduces the overhead and performance impact of a degraded state. Erasure coding example where N = 7 and K = 2 ## Choice of "N" Earlier, it was taken as a given that `N` is equal to 10. The exact choice of this value is somewhat arbitrary but is generally bounded by system-level consensus constraints as well as the calculated cost of the network transit during rebuild operations. As `N` increases, the storage overhead for the data reliability for some specified `K` *decreases* because `(N+K)/N` approaches 1. However, the total amount of information that must be considered and moved over the network for, as well as the actual computational cost of, a rebuild *increases*. Real-world usage has found that an `N` somewhere between 8 and 12 provides a good tradeoff between these two opposing considerations. ## NVMe Userspace Optimizations A final important aspect of the Ocient storage and I/O layer is the database engine’s direct interaction with the SSDs storing segments. The term "direct" in this context can be contrasted with how a more standard approach might look. In many database systems, record storage is achieved using a set of files stored on a disk in some file system (such as EXT4, NTFS, or ZFS). And generally speaking, kernel-provided system invocations or services, like `read()`, `write()`, and `mmap()`, perform reading and writing of those files. Obviously, this is a fine approach in general, but when performance is of utmost importance, the overhead of the system calls, memory copies to and from kernel memory, as well as the implementation of the underlying file systems, all have detrimental and sometimes unpredictable effects. For example, it is well known that the kernel of EXT4 implementation takes global locks during certain metadata operations, which can have a significant detrimental effect on parallelism and throughput. Some databases eschew standard files and perform some form of I/O directly against kernel block devices in order to avoid file system overhead. This is surely an improvement, but the kernel’s block scheduler, extra memory copies to/from kernel memory, and the overhead of system calls still exist. In order to overcome these performance impacts, the Ocient storage layer directly interacts with the NVMe SSDs that store segments in two important ways: 1. Each NVMe drive is decoupled from the standard Linux block device driver and instead attached to a device driver that enables low-level interaction with the device using the PCIe protocol. The implications of this approach are that the storage layer and database engine can directly communicate the NVMe protocol to the devices *and* do so using user memory. This completely eliminates all system calls and memory copies required to do I/O with the NVMe drives. The database provides direct pointers to its own virtual memory for I/O operations, and the drives directly write and read to and from that memory. An interesting result from this approach is that the Ocient storage layer and engine were entirely unaffected by the Linux kernel’s mitigations for the Spectre and Meltdown classes of attacks. This is because the primary mechanism of the mitigation resulted in a substantial increase in the overhead of the user-to-kernel transition associated with system calls. Other database engines with more standard approaches saw, in many cases, appreciable performance degradation. 2. Modern NVMe SSDs are capable of a million 4KiB random reads per second. This is strikingly and substantially more than HDDs (where even the best drives can produce only hundreds). However, for most economical SSDs, these rates can only be achieved when large numbers of parallel in-flight requests are maintained over a substantial fraction of the drive’s address space. The Ocient storage layer implementation, in conjunction with specially tailored data and metadata structures, not only promotes this parallelism to be evoked during query execution, but also is written to minimize "latency bubbles" by always having a replacement I/O operation teed up for a drive when it completes a previous one. Optimized use of NVMe SSDs maximizes I/O throughput ## Columnar Compression As a column-oriented data warehouse, Ocient is able to leverage per-column data locality and similarity to achieve excellent data compression. * For fixed-length columns, a combination of delta-delta compression, run-length encoding, and NULL elimination provides significant storage reduction for many data types. * For variable-length columns, the Ocient System can apply per-row compression to reduce the size of large values. * Both fixed and variable-length columns can benefit from whole-column compression using a shared dictionary trained on a per-column and per-segment basis. Column compression works best when combined with secondary indexes on frequently filtered columns. ## Related Links [Secondary Indexes](/secondary-indexes) [Key Concepts](/key-concepts) ## Related Videos [At the Whiteboard with Ocient: Compute Adjacent Storage Architecture™](https://www.youtube.com/watch?v=7cv0hr7f1fg) *** *Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.* # Configuration Settings for Data Pipelines Source: https://docs.ocient.com/configuration-settings-for-data-pipelines Reference for Ocient data pipeline configuration settings, including JVM memory, S3 timeouts, Kafka polling, Parquet memory limits, and file system access. Data pipeline functionality enables the use of SQL statements to load data. You can use a variety of configuration settings to manage the load. See this table for the settings that you can use. | **Configuration Setting Parameter Name** | **Default** | **Data Type** | **Description** | | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `streamloader.extractorEngineParameters.enabled` | `true` | BOOLEAN | Whether the Loader Node attempts to bootstrap the extractor process for data pipelines on start-up. If you set this parameter to `true`, the Loader Node does not become active until the Java loading process has successfully started. | | `streamloader.extractorEngineParameters.jvmMemoryConfiguration` | - `32g` for `initialHeap`
- `64g` for `maxHeap`
- `64g` for `maxDirect` | VARCHAR | The memory configuration to use for the extractor Java process. Use strings representing data sizes in SI format: 1 KiB, 5 MiB, etc.
You must set the three memory options in the same statement.

**Example**
`ALTER SYSTEM ALTER CONFIG SET 'streamloader.extractorEngineParameters.jvmMemoryConfiguration.initialHeap' = '32g',`
`'streamloader.extractorEngineParameters.jvmMemoryConfiguration.maxHeap' = '64g',`
`'streamloader.extractorEngineParameters.jvmMemoryConfiguration.maxDirect' = '64g';` | | `streamloader.extractorEngineParameters.restPort` | 8090 | INT | Use this parameter with the `streamloader.extractorEngineParameters.portOffset` parameter to form the REST port for the extractor. (External monitoring services like can query the REST server for loading process metrics.) | | `streamloader.extractorEngineParameters.portOffset` | 0 | INT | Use this parameter with the `streamloader.extractorEngineParameters.restPort` parameter to form the REST port of the extractor. (External monitoring services like Telegraf can query the REST server for loading process metrics.) | | `streamloader.extractorEngineParameters.configurationOption.expect.empty.file.list` | `false` | BOOLEAN | Set this value to `false` to fail a file source data pipeline that finds no files to load instead of the load completing successfully with no data loaded. | | `streamloader.extractorEngineParameters.configurationOption.pipeline.preview.rows.limit` | 1000 | INT | The maximum number of rows a `PREVIEW PIPELINE` SQL statement can return. For example, when you use this statement, if the specified `LIMIT` value exceeds this number, then the loading process throws an error. | | `streamloader.extractorEngineParameters.configurationOption.engine.transform.udt.jarRootDirectory` | `/opt/ocient/current/lib/extractorengine_udt` | VARCHAR | The absolute path to the directory containing JARs for data pipeline functions. To install and enable a third-party library for using data pipeline functions, install the JAR package at this location on all Loader Nodes. Then, add the chosen fully-qualified class name to the IMPORT clause of the data pipeline function. For details, see [CREATE OR REPLACE PIPELINE FUNCTION](/data-pipelines#create-pipeline-function). | | `streamloader.extractorEngineParameters.configurationOption.engine.external.jdbc.jarRootDirectory` | `/opt/ocient/current/lib/extractorengine_jdbc` | VARCHAR | The absolute path to the directory containing JARs for external source lookup functionality. For details, see [Load Data from External Sources in Data Pipelines](/load-data-from-external-sources-in-data-pipelines). | | `streamloader.extractorEngineParameters.configurationOption.source.record.max.size` | Dynamically calculated value at start-up based on the number of processors and amount of memory available to the JVM | BIGINT | The maximum source record size, in bytes, that the loading process tolerates before throwing an error. | | `streamloader.extractorEngineParameters.configurationOption.s3.region` | `us-east-1` | VARCHAR | The default region for S3 file sources. | | `streamloader.extractorEngineParameters.configurationOption.s3.force.path.style.access` | `false` | BOOLEAN | Set this value to `true` to use path-style access for all S3 file sources. The default is a virtual-hosted style. | | `streamloader.extractorEngineParameters.configurationOption.s3.force.path.style.access.if.endpoint.overriden` | `true` | BOOLEAN | Set this value to `true` to use path-style access for S3 file sources that specify endpoints. (The system still uses virtual-hosted style access for sources without endpoint overrides unless you set `streamloader.extractorEngineParameters.configurationOption.s3.force.path.style.access` to `true`). | | `streamloader.extractorEngineParameters.configurationOption.s3.retries.count` | 10 | INT | The number of retries for the S3 file source client. | | `streamloader.extractorEngineParameters.configurationOption.s3.netty.read.timeout.seconds` | 0 | INT | The timeout value for read operations, in seconds, for the S3 file source client. `0` means do not perform a timeout. | | `streamloader.extractorEngineParameters.configurationOption.s3.netty.max.concurrency` | 50 | INT | Maximum number of allowed concurrent requests for the S3 file source client. | | `streamloader.extractorEngineParameters.configurationOption.awssdk.backoff.strategy.base.delay.seconds` | 1 | INT | The base amount of time, in seconds, for calculating the time the S3 file source client waits before retrying a failed request. The range of values that the calculated time can achieve increases exponentially with each failure. You can set the maximum value using the `streamloader.extractorEngineParameters.configurationOption.awssdk.backoff.strategy.max.backoff.seconds` parameter. | | `streamloader.extractorEngineParameters.configurationOption.awssdk.backoff.strategy.max.backoff.seconds` | 20 | INT | The maximum amount of time the S3 file source client waits before retrying a failed request. | | `streamloader.extractorEngineParameters.configurationOption.awssdk.max.pending.connection.acquires` | 20000 | INT | The maximum number of pending connection acquires that the S3 file source client allows. | | `streamloader.extractorEngineParameters.configurationOption.awssdk.connection.timeout.seconds` | 10 | INT | The amount of time, in seconds, that the S3 file source client waits when initially establishing a connection. | | `streamloader.extractorEngineParameters.configurationOption.awssdk.connection.max.idle.timeout.seconds` | 300 | INT | The maximum amount of time, in seconds, that the system allows a connection established by the S3 file source client to remain open while idle. | | `streamloader.extractorEngineParameters.configurationOption.awssdk.connection.acquisition.timeout.seconds` | 300 | INT | The amount of time, in seconds, that the S3 file source client waits when acquiring a connection from the pool. | | `streamloader.extractorEngineParameters.configurationOption.kafka.poll.timeout.msec` | 1000 | BIGINT | The amount of time, in milliseconds, that a consumer for a pipeline with an source waits for records to become available during a poll operation. | | `streamloader.extractorEngineParameters.configurationOption.kafka.assign.timeout.msec` | 20000 | BIGINT | The amount of time, in milliseconds, that a `PREVIEW PIPELINE` SQL statement waits for partitions to be assigned to the consumer on a pipeline with a Kafka source. If the system does not assign any partitions when the timeout elapses, then the `PREVIEW PIPELINE` statement terminates without data processing. | | `streamloader.extractorEngineParameters.configurationOption.kafka.activate.timeout.msec` | 5000 | BIGINT | The amount of time, in milliseconds, that a `PREVIEW PIPELINE` SQL statement waits for a partition to receive records on a pipeline with a Kafka source. | | `streamloader.extractorEngineParameters.configurationOption.arrow.max.native.memory.usage.bytes` | 23756537856 | BIGINT | The default maximum native memory usage, in bytes, for the reader. | | `streamloader.extractorEngineParameters.configurationOption.arrow.max.java.buffer.memory.usage.bytes` | 15837691904 | BIGINT | The default maximum buffer memory usage, in bytes, for the Parquet reader. | | `streamloader.extractorEngineParameters.configurationOption.filesystem.access.directories` | `/tmp` | VARCHAR | Comma-separated list of directories where you can load server file system data.

Each directory must exist.

All files must be located within one of the directories.

By default, you can load from the `/tmp` directory.
| ## Related Links [Load Data](/load-data) [Alter Default Data Pipeline Behavior](/alter-default-data-pipeline-behavior) # Configure Data Pipeline Logging Source: https://docs.ocient.com/configure-data-pipeline-logging Adjust log levels for Ocient data pipelines with the ALTER SYSTEM ALTER LOG LEVEL SQL command to troubleshoot loading issues without impacting performance. You can troubleshoot data pipelines using the standard logging mechanism in . Set these log levels in consultation with Ocient Support because this setting might impact performance. ## Modify the Level of Logging Execute these two SQL statements in this order to bump up the log message. First, set the log level. ```sql SQL theme={null} ALTER SYSTEM ALTER LOG LEVEL SET 'log4j' 'EDEBUG'; ``` Then, set the level for the logging configuration. ```sql SQL theme={null} ALTER SYSTEM ALTER CONFIG SET 'streamloader.extractorEngineParameters.loggingConfig.com.ocient.extractor.allowLevel' 'trace'; ``` If the log level of the first SQL statement is lower than the log level of the second SQL statement, the system loses log entries. It is also possible to repeat the second SQL statement for different package names, such as: * `ALTER SYSTEM ALTER CONFIG SET 'streamloader.extractorEngineParameters.loggingConfig.com.ocient.extractor.allowLevel' 'Info';` * `ALTER SYSTEM ALTER CONFIG SET 'streamloader.extractorEngineParameters.loggingConfig.com.ocient.extractor.en gine.allowLevel' 'Trace';` ## Valid Log Levels This table provides an overview of the log levels you can specify for the `ALTER SYSTEM ALTER LOG LEVEL SET` SQL statements with the `streamloader.extractorEngineParameters.loggingConfig` keys. The necessary log level for `log4j` must be set accordingly to at least the value specified in the column for the `log4j` level. | **Log Level of** `streamloader.extractorEngineParameters.loggingConfig` | **Derived Level for** `log4j` | | ----------------------------------------------------------------------- | ----------------------------- | | FATAL | ERROR | | ERROR | ERROR | | WARN | WARN | | INFO | INFO | | DEBUG | DEBUG | | TRACE | EDEBUG | ## Reset the Log Level To reset the log level after you reproduce a problem, execute these SQL statements. ```sql SQL theme={null} ALTER SYSTEM ALTER CONFIG RESET 'streamloader.extractorEngineParameters.loggingConfig.com.ocient.extractor.allowLevel'; ALTER SYSTEM ALTER CONFIG RESET 'log4j'; ``` ## Related Links [Monitor Data Pipelines](/monitor-data-pipelines) [Data Pipeline Loading Errors](/data-pipeline-loading-errors) # Configure Storage Spaces Source: https://docs.ocient.com/configure-storage-spaces Configure storage spaces in an Ocient System, including segment group width, parity width, and overprovisioning for fault tolerance and capacity planning. When setting up an System, a storage cluster represents a set of Foundation Nodes in the system. A storage space is a set of parameters within a storage cluster that defines how the system administers data and fault tolerance. Creating a storage space is an important step for launching an Ocient System that must be done after installing hardware and bootstrapping nodes, but before you start loading and querying data. To see how creating a storage space fits into the Ocient installation process, see [Ocient Application Configuration](/ocient-application-configuration). This tutorial explains how to set up and configure storage spaces to meet the needs of your Ocient System. OcientAIQ Unified Data Platform contains Foundation Nodes in segments grouped into segment groups that are grouped into storage clusters. Tables in schemas with metadata are stored in segments. When an Ocient System starts, the system automatically creates a default system storage space for persistent metadata, such as system catalog tables. This default storage space is immutable, meaning you cannot delete or modify it. The name of the metadata storage space is `systemStorageSpace` in the [sys.storage\_spaces](/system-catalog#sys-storage_spaces) system catalog table. ## User-Defined Storage Spaces For storage other than metadata, you must create one or more separate storage spaces. The creation of storage spaces is necessary before loading data or performing other operations. User-defined storage spaces are configurable for extra resiliency or storage based on your system needs. A storage space represents how the storage cluster spreads data across Foundation Nodes to balance storage and fault tolerance. In general terms, the storage cluster configuration defines how the storage cluster behaves for these attributes: * Regular data storage operations (see [Segment Group Width](#segment-group-width)) * Query resiliency (see [Parity Width](#parity-width)) * Load resiliency (see [Overprovision](#overprovision)) You can configure a storage space to assign nodes to these job roles by using DDL statements (see [CREATE STORAGESPACE](/cluster-and-node-management#create-storagespace)). Planning out your storage space carefully is important because you cannot change a storage space configuration after creation. ### **Segment Group Width** Storage space parameter: `WIDTH` The Segment Group Width determines the number of segments that comprise a segment group. Functionally, this setting defines how many Foundation Nodes perform read and write operations for a segment group. Within a Segment Group, the system assigns each segment to a different node. Hence, the Segment Group Width cannot exceed the number of nodes in your system. In most circumstances, Segment Group Width should comprise most of your Foundation Nodes. On system startup, the default Segment Group Width is three, which is the minimum number of nodes required for an Ocient System. ### Parity Width Storage space parameter: `PARITY_WIDTH` You can assign a subset of the Segment Group Width to Parity Width. The Parity Width of a storage space determines its fault tolerance, defining the number of parity coding bits to use for each segment group. This number determines how many nodes can fail before the cluster is disabled. Parity Width protects system querying capabilities. If any nodes fail, the system can execute and complete queries as long as the number of failed nodes is less than or equal to the Parity Width. When deployed in conjunction with overprovisioned nodes, Parity Width also provides fault tolerance for loading operations. The `PARITY WIDTH` requires additional storage overhead, which is calculated by the formula `(PARITY WIDTH) / (SEGMENT GROUP WIDTH - PARITY WIDTH)`. ### Overprovision The Ocient System overprovisions any Foundation Nodes in the cluster in excess of the Segment Group Width by default. These nodes can include any nodes not configured in the storage space or any nodes added later. Overprovisioned nodes protect loading operations from node failure, although they must be used in conjunction with Parity Width. In the event of node failures, loading operations can continue as long as the number of failed nodes does not exceed the Parity Width nodes or the number of overprovisioned nodes. For this reason, providing fault tolerance for loading requires a balance of Parity Width and overprovisioned nodes. Unlike Segment Group Width and Parity Width, you do not explicitly define a value for overprovisioning. Instead, any Foundation Nodes in the cluster in excess of the number allocated toward Segment Group Width or Parity Width become overprovisioned by default. ## Storage Space Configuration Examples This section demonstrates different configurations for storage spaces. The examples each go through different scenarios for node failures to show the fault tolerance of each setup. For information on the syntax for storage spaces, see [CREATE STORAGESPACE](/cluster-and-node-management). ### 10-Node Cluster This example assumes you have a storage cluster of 10 Foundation Nodes. ```sql SQL theme={null} CREATE STORAGESPACE ocient WIDTH = 10, PARITY_WIDTH = 2; ``` * The `WIDTH = 10` parameter means the system stores a segment group on 10 different nodes. * The `PARITY_WIDTH = 2` parameter means each segment in the group contains enough parity bits to restore lost data for up to two nodes. In effect, this means parity bits comprise 20 percent of storage. **Tolerance Scenarios** This storage space setup has no overprovisioning, making loading operations less resilient. This setup results in these outcomes if nodes become disabled: * If one node fails, loading fails, and querying can continue. * If two nodes fail, loading fails, and querying can continue. * If three nodes fail, loading and querying both fail. ### 12-Node Cluster This example assumes a storage cluster of 12 Foundation Nodes. ```sql SQL theme={null} CREATE STORAGESPACE ocient WIDTH = 10, PARITY_WIDTH = 2; ``` Note that this DDL statement is the same as the [10-Node Cluster Example](#10-node-cluster), but the cluster in this example has two additional nodes. * The `WIDTH = 10` parameter means the system stores a segment group on 10 of the 12 nodes. * The two remaining nodes are overprovisioned. * The `PARITY_WIDTH = 2` parameter means each segment in the group contains enough parity bits to restore lost data for up to two nodes. In effect, this means parity bits comprise 16.6 percent of storage. **Tolerance Scenarios** This setup provides a level of resiliency for both querying and loading. The setup results in these outcomes if nodes become disabled: * If one node fails, loading and querying continues. * If two nodes fail, loading and querying continues. * If three nodes fail, loading and querying both fail. ### 12-Node Cluster with More Parity Width This example assumes a storage cluster of 12 Foundation Nodes with more nodes allocated to parity. ```sql SQL theme={null} CREATE STORAGESPACE ocient WIDTH = 10, PARITY_WIDTH = 3; ``` * The `WIDTH = 10` parameter means the system stores a segment group on 10 of the 12 nodes. * The two remaining nodes are overprovisioned. * The `PARITY_WIDTH = 3` parameter means each segment in the group contains enough parity bits to restore lost data for up to three nodes. In effect, this means parity bits comprise 42 percent of storage. **Tolerance Scenarios** This setup provides resiliency for loading and especially for querying. The setup results in these outcomes if nodes become disabled: * If one node fails, both loading and querying continue. * If two nodes fail, both loading and querying continue. * If three nodes fail, loading fails, and querying continues. * If four nodes fail, both loading and querying fail. ### 15-Node Cluster This example assumes a cluster of 15 Foundation Nodes. The configuration in this setup includes five overprovisioned nodes, exceeding the Parity Width nodes. This number of overprovisioned nodes would be inefficient and unable to provide the full benefit in a real system, but this example demonstrates what happens if you added extra nodes at a later time to a cluster, which the storage space would recognize as overprovisioned. ```sql SQL theme={null} CREATE STORAGESPACE ocient WIDTH = 10, PARITY_WIDTH = 3; ``` * The `WIDTH = 10` parameter means the system stores a segment group on 10 of the 15 nodes. * The five remaining nodes are overprovisioned. * The `PARITY_WIDTH = 3` parameter means each segment in the group contains enough parity bits to restore lost data for up to three nodes. In effect, this means parity bits comprise 20 percent of storage. **Tolerance Scenarios** This setup provides resiliency for both querying and loading, but having more than three overprovisioned nodes provides no benefit because it exceeds three Parity Width nodes. This setup results in these outcomes if nodes become disabled: * If one node fails, loading and querying continue. * If two nodes fail, loading and querying continue. * If three nodes fail, loading and querying continue. * If four nodes fail, loading and querying both fail because this breaches the Parity Width tolerance. ## Related Links [Core Elements of an Ocient System](/core-elements-of-an-ocient-system) [Cluster and Node Management](/cluster-and-node-management) ## # Connect to Ocient Source: https://docs.ocient.com/connect-to-ocient Connect client applications to an Ocient SQL Node using JDBC, pyocient, or REST on the default port 4050 with automatic load balancing across nodes. You can connect to any of the SQL Nodes in an System to run queries or commands. The SQL Node is available by default on port `4050` at the IP address or hostname assigned to the SQL Nodes. When more than one SQL Node is present in the Ocient System, Ocient automatically loads balance connections between the nodes to even out the load across connections. Ocient provides a JDBC driver and a driver named `pyocient`. You can use any of these drivers to connect to an online Ocient System. To connect from a client computer that can access the SQL Node using the network, follow any of the following examples. Note that you must have the IP address of the SQL Node and a valid username and password to connect. * [Connect Using JDBC](/connect-using-jdbc) * [Connect Using pyocient](/connect-using-pyocient) If you are unable to connect, check the logs on the SQL Nodes for issues or check the [Errors and Warnings](/errors-and-warnings) for more detail on error codes. For supported third-party tools, see [Ocient Integrations](/ocient-integrations). # Connect Using JDBC Source: https://docs.ocient.com/connect-using-jdbc Learn how to set up and configure a JDBC connection to Ocient, ensuring secure and efficient data access for large-scale analytics. The JDBC driver can be executed in a command-line interface (CLI) mode or used directly in a program with standard JDBC methods. This example describes how to connect using the CLI mode. For details on using the JDBC driver in code, please see [Establishing a Connection](https://docs.oracle.com/javase/tutorial/jdbc/basics/connecting.html). ## Step 1: Download the JDBC Driver First, confirm the recommended version of the JDBC driver for your version of by finding it in [Version Compatibility](/version-compatibility). Download the correct JDBC driver from the [repository](https://mvnrepository.com/artifact/com.ocient/ocient-jdbc4) by clicking on your preferred version, then selecting **view all**, and finally downloading the JAR file that is labeled with the pattern `ocient-jdbc4--jar-with-dependencies.jar`. Save this file to a convenient location on your client machine. ## Step 2: Run the JDBC CLI Program To run the JDBC CLI Program, you must have installed on your machine. For the version, see [Version Compatibility](/version-compatibility). Next, run the following command to start the JDBC CLI Program. Assuming your JAR stored at `~/ocient-jdbc4-jar-with-dependencies.jar`. ```shell Shell theme={null} java -classpath ~/ocient-jdbc4-jar-with-dependencies.jar com.ocient.cli.CLI ``` The system prompts you to enter the username and password. ```shell Shell theme={null} Username: test@system Password: fakepassword ``` **Example Response:** ```shell Shell theme={null} Ocient> _ ``` The Ocient CLI prompt appears. ## Step 3: Connect To connect, enter the connection string to your SQL Nodes. SQL Nodes install by default with a self-signed TLS certificate for encryption by default, and a custom certificate can be present. Depending on the TLS configuration, you can use `tls=unverified` (default) or `tls=on` for the required setting. Assuming the standard port `4050`, a self-signed certificate, and the SQL Node IP Address of `10.10.1.1`, you can connect to the system database with the following connecting string: ```shell Shell theme={null} Ocient> connect to jdbc:ocient://10.10.1.1:4050/system; ``` **Example Response:** ```shell Shell theme={null} Connected to jdbc:ocient://10.10.1.1:4050/system Ocient> _ ``` You are now connected and can execute queries based on your user’s permissions. Try running: `select * from sys.tables;` to see what tables are defined. For more details on using the JDBC driver and the CLI program, see the [JDBC Manual](/jdbc-manual). The system cancels any queries made with the JDBC driver if you perform any of these actions: * Fetch the entire result set. * Close the connection. * Execute a new query on the same connection. ## Related Links [Connect to Ocient](/connect-to-ocient) [Ocient Integrations](/ocient-integrations) # Connect Using pyocient Source: https://docs.ocient.com/connect-using-pyocient Follow this guide to establish a Pyocient connection to the Ocient data warehouse, enabling Python-based data operations and analysis. The module `pyocient` supports direct connections to Ocient Systems in a Command-Line Interface (CLI) as well as in Python programs that follow the Python Database API Specification 2.0. After you install `pyocient`, you can establish Ocient connections to run queries and commands. ## Installation Tutorial To install and run `pyocient`, follow these steps. Install `pyocient` using pip. ```shell Shell theme={null} pip3 install pyocient ``` After running the pip install, `pyocient` is installed in the `bin` directory. Use `venv` to install `pyocient` using a virtual environment. For more information, see the Python [venv](https://docs.python.org/3/library/venv.html) documentation. `pyocient` is started from the command line with a connection string. It includes many options to configure the connection and client similar to the JDBC driver. Learn more about supported options in the [Ocient Python Module: pyocient](/ocient-python-module-pyocient). After installation, you can launch from a command line by typing the `pyocient` command. ```shell Shell theme={null} pyocient ocient://example_user:example_password@10.10.1.1/system ``` To connect to the system database, replace the example username (`example_user`) and password (`example_password`) with your credentials, set your IP address, The connection example assume a SQL Node exists at `10.10.1.1`. The example uses the default port `4050`, and is set up with default self-signed certificates for TLS. **Example Response** ```SQL SQL theme={null} Ocient Database™ System Version: , Client Version > _ ``` Use the command `quit;` to exit the `pyocient` CLI interface. In the `pyocient` CLI, you can run any SQL query supported by your database. In this example, a query is selecting rows from the system tables: ```sql SQL theme={null} SELECT * FROM sys.tables; ``` You can also run queries from the command line with the connection string: ```shell Shell theme={null} pyocient ocient://example_user:example_password@10.10.1.1/system 'SELECT * from sys.tables' ``` **Example Response** ```SQL SQL theme={null} [ { "id": "359e21b1-770a-4860-aec9-f158d307f725", "name": "data_type_coverage", "schema": "loading", "database_id": "f827ff7c-dd94-43bf-8386-93d048344a32", "storage_space_id": "2ad08e7c-15d1-4861-b38b-256c4df4a191", "maximum_segment_size_gib": 4, "description": null, "streamloader_property_string": null, "created_at": "2023-08-08T21:22:49.622213" } ] ``` The system cancels any queries made with the `pyocient` driver if you perform any of these actions: * Fetch the entire result set. * Close the connection. * Execute a new query using the same connection. For more information and API reference material for pyocient, see the [Ocient Python Module: pyocient](/ocient-python-module-pyocient) page. ## Related Links [Connect to Ocient](/connect-to-ocient) [Ocient Integrations](/ocient-integrations) # Connection Driver Reference Source: https://docs.ocient.com/connection-driver-reference Reference for Ocient connection drivers, including JDBC, ODBC, pyocient, and the HTTP Query API, with version compatibility and integration guidance. includes multiple ways to connect and issue statements or queries to the database. The most common drivers used for communicating with Ocient are the JDBC driver and the pyocient module. Many third-party tools use the JDBC driver for business intelligence, visualization, and data exploration. These drivers can also be used in custom programs by leveraging the programming language’s appropriate mechanisms to use the driver, for example, using a JDBC driver directly in . * [JDBC Manual](/jdbc-manual) * [Data Extract Tool](/data-extract-tool) * [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program) * [JDBC Classes and Methods](/jdbc-classes-and-methods) * [JDBC Spark Connector](/jdbc-spark-connector) * [OCGraph Java Library](/ocgraph-java-library) * [Ocient Python Module: pyocient](/ocient-python-module-pyocient) * [OCGraph Python Library](/ocgraph-python-library) # Conversion Functions Source: https://docs.ocient.com/conversion-functions Reference for OcientGeo conversion functions that transform geospatial values between WKT, WKB, GeoJSON, and Ocient native point, linestring, and polygon types. conversion functions transform a specified operand to a different data type. ## ST\_ASBINARY Alias of ST\_ASWKB. Returns the well-known binary (WKB) representation of the specified geography. **Syntax** ```sql SQL theme={null} ST_ASBINARY(geo [, XDR_or_NDR ] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, or `POLYGON` | The geospatial object used to calculate the dimensions. | | `XDR_or_NDR` | `CHAR` | Optional. This argument determines whether the resulting WKB is in little endian (NDR) or big endian (XDR) format.
If you specify this argument, it must be either `XDR` or `NDR`. Otherwise, it defaults to `NDR`. | **Example** ```sql SQL theme={null} SELECT ST_ASBINARY(ST_POINT(2, 2), 'NDR'); ``` *Output*: `010100000000000000000000400000000000000040` ## ST\_ASGEOJSON Returns the GeoJSON representation of the specified geography using the [IETF standards](https://datatracker.ietf.org/doc/html/rfc7946). ```sql SQL theme={null} ST_ASGEOJSON( geo [, planar_conversion ] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, or `POLYGON` | The geospatial object used to create a GeoJSON representation.
ℹ️ The GeoJSON specification has certain requirements for a `LINESTRING` and `POLYGON`.
In keeping with this specification, a `POLYGON` instance that represents a `LINESTRING` results in a `LINESTRING` GeoJSON instance.
Similarly, a `POLYGON` instance that represents a `POINT` results in a `POINT` GeoJSON instance.
A `LINESTRING` instance that represents a `POINT` results in a `POINT` GeoJSON instance.
Unlike the GeoJSON specification, a specified `POLYGON` does not need to follow the Right-Hand Rule. If a specified `POLYGON` contains any holes that are not valid `POLYGON` objects, then the function omits those holes in the resulting GeoJSON instance. | | `planar_conversion` | `BOOLEAN` | Optional. A Boolean value that determines if the geodesic edges of the specified geography are converted to planar representation before the function applies the GeoJSON format. standards state that all GeoJSON objects are to be interpreted as explicitly planar. This argument allows preservation of the original geodesic shape.
If `planar_conversion` is specified `TRUE`, points will be added such that the resulting GeoJSON represents a planar shape within 10m of the original geospatial object. Additionally, if the original geospatial object crossed the international dateline, it will be split at the point where it crosses. This will generate a GeoJSON object of either `MultiLineString` or `MultiPolygon` type, depending on the original type of the geospatial object.
If you do not specify this argument, the value defaults to `FALSE`. | **Examples** This example converts a `POINT` value into a GeoJSON representation. ```sql SQL theme={null} SELECT ST_ASGEOJSON(ST_POINT(2, 2), false); ``` *Output*: ```Text Text theme={null} {"type":"Point","coordinates":[2,2]} ``` This example converts an empty `POINT` value. ```sql SQL theme={null} SELECT ST_ASGEOJSON(ST_POINTFROMTEXT('POINT EMPTY'), false); ``` *Output*: ```Text Text theme={null} {"type":"Point","coordinates":[]} ``` This example converts a `LINESTRING` value into a GeoJSON representation. ```sql SQL theme={null} SELECT ST_ASGEOJSON(ST_LINESTRING(ST_POINT(1.234, 2),ST_POINT(3.456, 5)); ``` *Output*: ```Text Text theme={null} {"type":"LineString","coordinates":[[1.234,2],[3.456,5]]} ``` This example again converts a `LINESTRING` value. ```sql SQL theme={null} SELECT ST_ASGEOJSON(ST_LINESTRING(ARRAY[ST_POINT(1.234, 2)]); ``` *Output*: ```Text Text theme={null} {"type":"Point","coordinates":[1.234,2]} ``` This example converts an empty `LINESTRING` value into a GeoJSON representation. ```sql SQL theme={null} SELECT ST_ASGEOJSON(ST_LINEFROMTEXT('LINESTRING EMPTY'), false); ``` *Output*: ```Text Text theme={null} {"type":"LineString","coordinates":[]} ``` This example converts a `POLYGON` value into a GeoJSON representation. ```sql SQL theme={null} SELECT ST_ASGEOJSON( ST_POLYGON( ARRAY [ST_POINT(1.234, 2),ST_POINT(3.456, 5),ST_POINT(5.678, 2),ST_POINT(1.234, 2)], ARRAY [ARRAY[ST_POINT(2,2.5),ST_POINT(2.1,2.6),ST_POINT(2.2,2.5),ST_POINT(2,2.5)], ARRAY [ST_POINT(3,2.5),ST_POINT(3.1,2.6),ST_POINT(3.2,2.5),ST_POINT(3,2.5)]]), false); ``` *Output*: ```Text Text theme={null} {"type":"Polygon","coordinates":[[[1.234,2],[3.456,5],[5.678,2],[1.234,2]],[[2,2.5],[2.1,2.6],[2.2,2.5],[2,2.5]],[[3,2.5],[3.1,2.6],[3.2,2.5],[3,2.5]]]} ``` This example again converts a `POLYGON` value. ```sql SQL theme={null} SELECT ST_ASGEOJSON( ST_POLYGON( ARRAY [ST_POINT(1.234, 2),ST_POINT(3.456, 5),ST_POINT(5.678, 2),ST_POINT(1.234, 2)], ARRAY [ARRAY[ST_POINT(2,2.5),ST_POINT(2.1,2.6),ST_POINT(2.2,2.5),ST_POINT(3,2.5)], ARRAY [ST_POINT(3,2.5),ST_POINT(3.2,2.5),ST_POINT(3,2.5)]]), false); ``` *Output*: ```Text Text theme={null} {"type":"Polygon","coordinates":[[[1.234,2],[3.456,5],[5.678,2],[1.234,2]]]} ``` This example converts a two-point `POLYGON`, which results in a `LINESTRING` type in the GeoJSON representation. ```sql SQL theme={null} SELECT ST_ASGEOJSON( ST_POLYGON( ST_LINESTRING(ST_POINT(1.234, 2), ST_POINT(3.456, 5))), false); ``` *Output*: ```Text Text theme={null} {"type":"LineString","coordinates":[[1.234,2],[3.456,5]]} ``` This example converts a single-point `POLYGON`, which results in a `POINT` type in the GeoJSON representation. ```sql SQL theme={null} SELECT ST_ASGEOJSON(ST_POLYGON(ST_POINT(1.234, 2)), false); ``` *Output*: ```Text Text theme={null} {"type":"Point","coordinates":[1.234,2]} ``` This example converts an empty `POLYGON` value into a GeoJSON representation. ```sql SQL theme={null} SELECT ST_ASGEOJSON(ST_POLYGONFROMTEXT('POLYGON EMPTY'), false); ``` *Output*: ```Text Text theme={null} {"type":"Polygon","coordinates":[]} ``` ## ST\_ASLATLONTEXT Returns a string that represents geographic coordinates of a specified `POINT` in the specified format. ```sql SQL theme={null} ST_ASLATLONTEXT(point, [ output_format ] ) ``` | **Argument** | **Data** **Type** | **Description** | | --------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `point` | `POINT` | A POINT value used to calculate geographic coordinates.
If `point` is NULL, then this function returns NULL. | | `output_format` | `CHAR` | Optional. A string that includes option flags to set the format and decimal precision for the output of the coordinates.
Option flags must be uppercase to be recognized. Accepted option flags are:
`D` — Represents coordinate degrees. If you specify a format string for `output_format`, this value is required. If `M` is omitted, this option flag sets the decimal precision.
`M` — Represents minutes of latitude or longitude distance. If `S` is omitted, this option flag sets the decimal precision.
`S` — Represents seconds of latitude or longitude distance.
`C` — Represents the cardinal direction. If you specify this value, the appropriate cardinal direction is represented as N/S/E/W. If you omit this value, coordinates that are south or west are represented as negative values while east or north remain positive values.
For all option flags besides `C`, you can repeat the option characters and use them with decimal points to set the format and decimal precision. The number of characters after the decimal point indicates the precision, while the number of characters before the decimal indicates the total width of the formatted number, including the decimal POINT.
If the formatted number is shorter than the requested width, the system pads the value with leading spaces.
Characters other than `D`, `M`, `S`, `C`, and `.` are passed through as string characters.
If you do not specify `output_format` or leave it as empty, the value defaults to the format `D°M''S.SSS"C`. | **Example** ```sql SQL theme={null} SELECT ST_ASLATLONTEXT(ST_POINT(10.2342342,-2.32498), 'D.D degrees MM.MM minutes C'); ``` *Output*: `'2.0 degrees 19.50 minutes S 10.0 degrees 14.05 minutes E'` ## ST\_ASTEXT Alias of ST\_ASWKT and ST\_EWKT. Returns the well-known text (WKT) representation of the specified geography. If you use `ST_ASEWKT`, the database prepends the resultant string with `SRID=4326;`. ```sql SQL theme={null} ST_ASTEXT(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `geo` | `POINT`, `LINESTRING`, or `POLYGON` | A geospatial object that is to be returned as a WKT value.
If you specify a `LINESTRING` or `POLYGON` that contains a single `POINT`, the function returns the WKT representation as a `POINT`.
If you specify a `POLYGON` that has an exterior that is not closed, the function returns the WKT representation as a `LINESTRING`. | **Example** ```sql SQL theme={null} SELECT ST_ASTEXT(ST_POLYGON(ST_LINESTRING('LINESTRING(1 2, 1 3, 1 3, 1 2)'))); ``` *Output*: `POLYGON((1 2, 1 3, 1 3, 1 2))` ## ST\_ASEWKT For usage, see [ST\_ASTEXT](#st_astext). ## ST\_ASWKT Alias of [ST\_ASTEXT](#st_astext). ## ST\_ASWKB Alias of [ST\_ASBINARY](#st_asbinary). ## ST\_GEOHASH Returns a string that represents the geohash of the input `POINT`. This function also accepts an optional second argument to specify the length of the geohash result. If you specify any argument as NULL, then the function returns NULL. ```sql SQL theme={null} ST_GEOHASH(point [, precision ] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `point` | `POINT` | A `POINT` value that is to be converted into a geohash. | | `precision` | `INTEGER` | Optional. A numeric value that specifies the geohash precision, which determines the length of the output string.
This number must be in the range of 1 to 20. If you do not specify this value, `precision` defaults to 20. | **Example** ```sql SQL theme={null} SELECT ST_GEOHASH(ST_POINT(-126, 48)); ``` *Output*: `c0w3hc0w3hf1s70w3hf1s70w3` ## Related Links [Geospatial Data Types](/data-types#geospatial-data-types) [Attribute Functions](/attribute-functions) # Core Elements of an Ocient System Source: https://docs.ocient.com/core-elements-of-an-ocient-system Explore the core elements of an Ocient System, including SQL Nodes, Foundation Nodes, Loader Nodes, storage clusters, and the global metadata services. ## The System and Nodes The System is the collection of all the nodes within a single environment. This architecture diagram is an example of one Ocient System. Across systems, the number of nodes can vary along with the total storage capacity. Within a single system, users can define varying numbers of databases, storage groups, users, etc. An Ocient System is made up of a collection of distributed nodes which each play a predefined role in the data warehouse. The different roles are generally responsible for storing and analyzing data, administration, and moving data into Ocient. These nodes are connected to each other through a high-speed 100 Gbps network connection. In addition to storing user data, the data warehouse also stores metadata about the system and data for its own internal operation and recovery. This collection of nodes, their data, and the interconnection between them make up the Ocient System. The end user, analyzing data in Ocient, connects to this system through a JDBC or a -based client. From there, they interact with the database objects, such as tables and views. The architecture itself is abstracted from users, allowing them to focus on SQL queries and results. While the number of nodes in an environment might vary, the inclusion of and purpose of the different roles do not. The following sections briefly describe the components in the architecture. Ocient architecture diagram that shows the relationship between data sources, loading and transformation of the data, and data storage ### Node Types #### SQL Nodes A SQL Node is responsible for receiving incoming SQL requests and parsing the statements. This node also serves as the interface for System Administrators and Database Administrators to configure and maintain an Ocient System. Administrators can connect to a SQL Node using the Ocient command-line interface or their chosen client. When a SQL Node receives a query, it creates a plan for it using one of two optimization methods. After there is a plan for the query, the SQL Node distributes it to the Foundation Nodes. When the Foundation Nodes finish their processing, they return data back to the SQL Node for further processing and packaging the result set to the client. When the result set is finalized, the SQL Node returns the data to the client. Beyond the Foundation Nodes, further processing is done on the SQL Nodes to process intermediate result sets delivered by the Foundation Nodes. Depending on the query, the SQL Node is responsible for a different proportion of the overall work of the query. Common processing done here are aggregations and joins that happen across the intermediate results delivered by the Foundation Nodes. Administrators also use DDL or DCL statements to make changes to an Ocient System. When submitted to a SQL Node, the node executes the SQL statement across the system or forwards the SQL statement to the node assigned to execute it. #### Foundation Nodes The Foundation Nodes store the user data in Ocient and perform the bulk of query processing. A key architectural concept of Ocient is the , which collocates data and processing where possible. The Foundation Node is the central element of this architectural principle. When a query is deployed to the Foundation Node, it performs as much of the query as it can with the data on the node before having to join, aggregate, or compare it to other data. It returns that result up to the SQL Node for further processing, packaging, and returning to the user. Foundation Nodes contain the majority of the storage in an Ocient System and are also typically the largest in number. #### Loader Nodes Loader Nodes are responsible for extracting, transforming, indexing, and loading data ingested by Ocient from batch file sources as well as streaming data sources such as . A user can specify the extraction source details and supply a transformation pipeline that manipulates the structured or semi-structured source data before loading it into relational form into Ocient tables. Loader Nodes operate in a horizontal scale-out fashion allowing the loading system to scale to fit various ingestion requirements. Transparently to the end user, the Loader Node also manages exactly-once guarantees with data as it is loaded and converts pages into segments in the Foundation Nodes. ## Flows and Networking Between Nodes Data and communication flows across the nodes of an Ocient System for a variety of different purposes. There are two networks that connect nodes, a 100 Gbps high speed network and a 10 Gbps network. These are used in different ways for data flows and administrative purposes. ### A Query in Ocient When a user issues a query against the database, their SQL statement moves from the JDBC client to the SQL Node. The SQL Node parses and optimizes the query before handing the plan off to the Foundation Nodes. The Foundation Nodes do the first level of processing against the data before handing the results back to the SQL Node. The SQL Node further processes the data, constructs the results, and returns them to the client. * The connection between the SQL Nodes and Foundation Nodes is typically a 100 Gbps network connection. The speed of the connection between the client and the SQL Node is subject to the speed of that network (outside Ocient). * Every Foundation Node is connected to every SQL Node. ### Administration Flows When administrators configure or make changes to an Ocient System, they do it using DDL and DCL using a SQL client. The SQL client relays the statement to a SQL Node, which parses it into commands that the Administrator Role on the SQL Node executes. This can impact the SQL Nodes, Foundation Nodes, or Loader Nodes, depending on the type of change or operation. * The administration flow connections with the Ocient System are 10 Gbps connections. * Every node is connected to the SQL Nodes that perform the Administrator Role. ### Loading Flow When the load job is defined and started, the Loader Nodes read data from the source files, stage the data, process that data, and move the data into the Foundation Nodes. Data pipelines enable you to define the data load and any corresponding transformations to ensure data is properly loaded. * The network connections between the Loader and the Foundation Nodes are 100 Gbps. * The Loader Nodes connect to every Foundation Node. ## Database, Tables, and Views Similar to other databases, an is a grouping of databases, schemas, tables, views, users, and data within one Ocient System. A single system can have multiple databases and schemas. Schemas define the structure of a database and contain the definition of tables, views, and indexes. While Ocient stores data on disk in columnar format, creating and working with tables in Ocient is like any other relational database. Ocient supports a wide variety of standard SQL data types along with an expanding set of geospatial data types. Read more about: * [Data Definition Language (DDL) Statement Reference](/data-definition-language-ddl-statement-reference) * [Data Query Language (DQL) Statement Reference](/data-query-language-dql-statement-reference) * [Functions Overview](/functions-overview) ## TimeKey and Clustering Key Each table in Ocient can contain a column that is specified when defining the table. The TimeKey column is used to partition the data within the Foundation Nodes. Time partitioning is an important performance mechanism used by the OcientAIQ Unified Data Platform. Because many, if not most, queries specify some time filter for the results, time partitioning allows Ocient to quickly skip data from irrelevant times. When defining a table, users can also specify a clustering key containing one or more of the columns to further subdivide records on disk. This allows the database to quickly find records with the same key values. Read more in [TimeKeys and Clustering Keys](/timekeys-and-clustering-keys). ## Resilience to Hardware Failure In order to provide reliability in the case of a hardware outage, Ocient uses erasure coding. Erasure coding is a mechanism to organize and compute parity blocks so that the system can rebuild missing data. This means that Ocient does not store redundant copies of the data for reliability. As a result, an Ocient System requires less overall storage than if it were storing one or more copies of the data for failover. When a user defines their storage configuration, they specify the width of the group and the parity width. These values can vary based on how many Foundation Nodes are in the system and how resilient users want to make the Ocient System. ## Segments and Segment Groups Segments in Ocient are storage units that contain rows and are divided into a fixed number of coding blocks with a defined size. Segments are typically sized on the order of gigabytes. You set the segment size when you define tables in Ocient. The size of the segment must be a multiple of the coding block size. The coding block is the unit of parity calculation and the smallest unit of recovery as well. Multiple segments combine to form Segment Groups. A Segment Group has a fixed number of segments, named the width, which is the number of segments in the group. It also has a pre-defined number of parity blocks per set of data blocks for resiliency. Each segment has a defined index in the group. The Segment Group is physically stored in a storage cluster. Segment Groups can also be nested in directories named Segment Directory Groups. When the user configures an Ocient System, they need to define at least one storage space and a storage cluster. A storage cluster is a set of Foundation Nodes with an associated storage space. Also, these nodes coordinate together to store segment groups in a reliable manner. The storage cluster also has a width in the number of nodes. The width of the cluster and the Segment Groups stored on it must be the same. At the storage space level, the user defines the width and parity width of the space. Segments and their Segment Groups are stored across these storage spaces and clusters. To learn more about erasure coding, see [Compute-Adjacent Input and Output on Large Working Sets](/compute-adjacent-input-and-output-on-large-working-sets#erasure-coding). For details, see [Configure Storage Spaces](/configure-storage-spaces). OcientAIQ Unified Data Platform contains Foundation Nodes in segments grouped into segment groups that are grouped into storage clusters. Tables in schemas with metadata are stored in segments. ## Related Links * [Ocient Architecture](/ocient-architecture) * [Compute-Adjacent Input and Output on Large Working Sets](/compute-adjacent-input-and-output-on-large-working-sets) * [Key Concepts](/key-concepts) ## Related Videos * [At the Whiteboard with Ocient: Deployment Options](https://youtu.be/gkTb4dRHNKc) * [At the Whiteboard with Ocient: Data Reliability](https://www.youtube.com/watch?v=MKR8j9kYXvg) # CREATE TABLE SQL Statement Examples Source: https://docs.ocient.com/create-table-sql-statement-examples Examples of the Ocient CREATE TABLE SQL statement showing time-keyed tables, clustering keys, secondary indexes, partitioning, and supported data types. Tables in the contain various configuration options that can impact performance, reduce storage requirements, or provide extra data safeguards. As some table configuration options cannot be added after data is already loaded, it is important that you take the time to design `CREATE TABLE` SQL statements with forethought for the queries that you expect to execute frequently. This topic provides examples of `CREATE TABLE` SQL statements designed for different use cases and describes the configuration choices. For syntax and parameter information, see [CREATE TABLE](/tables#create-table). ## Create a Table with All Data Types This example creates a table with all supported data types in . The intent of this example is to show the required formatting for different data type values, which is shown as the `DEFAULT` value for each column. ```sql SQL theme={null} CREATE TABLE data_type_example ( col_bigint BIGINT NOT NULL DEFAULT 9876543210, col_binary BINARY(3) NOT NULL DEFAULT '0xabcdef', col_boolean BOOL NOT NULL DEFAULT TRUE, col_char CHAR(4) NOT NULL DEFAULT 'val', col_date DATE NOT NULL DEFAULT '2000-01-01', col_decimal DECIMAL(18,4) NOT NULL DEFAULT 123.45, col_double DOUBLE NOT NULL DEFAULT 3.141592, col_float FLOAT NOT NULL DEFAULT 2.718, col_int INT NOT NULL DEFAULT 123456789, col_int_array INT[] NOT NULL DEFAULT 'INT[0,1,2,3]', col_ipv4 IPV4 NOT NULL DEFAULT '127.0.0.1', col_ip IP NOT NULL DEFAULT '0123:4567:89ab:cdef:0123:4567:89ab:cdef', col_smallint SMALLINT NOT NULL DEFAULT 32767, col_matrix MATRIX[2][3] NOT NULL DEFAULT '{ {0, 0, 0}, {0, 0, 0} }', col_point POINT NOT NULL DEFAULT 'POINT(0 0)', col_linestring LINESTRING NOT NULL DEFAULT 'LINESTRING(0 0, 1 1)', col_polygon POLYGON NOT NULL DEFAULT 'POLYGON((0 0, 0 1, 1 1, 1 0, 0 0))', col_time TIME NOT NULL DEFAULT '12:34:56.012345678', col_timestamp TIMESTAMP NOT NULL DEFAULT '2000-01-02 12:34:45', col_tinyint TINYINT NOT NULL DEFAULT 127, col_tuple TUPLE<> NOT NULL DEFAULT 'tuple<>(0,0)', col_array_of_tuples TUPLE<>[] NOT NULL DEFAULT 'TUPLE<>[tuple<>(1,2),tuple<>(3,4)]', col_uuid UUID NOT NULL DEFAULT '01234567-89ab-cdef-1357-0123456789ab', col_varbinary VARBINARY NOT NULL DEFAULT '0xaabbccddeeff', col_varchar VARCHAR NOT NULL DEFAULT 'This is a variable length string' ); ``` ## Create a Table for Real-Time Analysis This example assumes you plan to ingest large quantities of time series data for rapid querying. The most important columns that you plan to reference in queries frequently include: * `created_at` — A granular timestamp column that represents the primary time reference in the table. * `user_id` — An integer column to identify each unique user. * `sell_id` — An integer column representing each transaction by a user. ```sql SQL theme={null} CREATE TABLE IF NOT EXISTS "transact_data" ( "created_at" TIMESTAMP TIME KEY BUCKET(1, HOUR) NOT NULL, "user_id" BIGINT NOT NULL DEFAULT 0, "sell_id" BIGINT NOT NULL, "purchase_amount" DOUBLE NOT NULL DEFAULT 0, "buyer_name" VARCHAR, CLUSTERING KEY "ck" ("sell_id", "user_id") ); CREATE INDEX "purchase_idx" ON "transact_data" ("purchase_amount"); CREATE INDEX "buyer_idx" ON "transact_data" ("buyer_name") USING NGRAM(4); ``` This example table makes use of these Ocient System configurations. | **Column Name** | **Configuration Description** | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `created_at` | This column is the table , which means that the database indexes for filter optimization.
The example configures the TimeKey to partition segments by every hour, `(1, HOUR)`, which is the approximate time granularity you expect to use in your query filters. | | `user_id`, `sell_id` | The example table identifies these two columns as the table Clustering Key. The database pairs these two columns for quick reference, such as filtering one column for row values of the other. | | `purchase_amount` | The `CREATE INDEX` SQL statement adds a secondary index to this column.
As a `BIGINT` data type, the `purchase_amount` column defaults to the `INVERTED` index. | | `buyer_name` | This `CREATE INDEX` SQL statement adds a secondary index to this column.
As a `VARCHAR` column, the `buyer_name` column normally defaults to a `HASH` index, but this example specifies it should be an `NGRAM` index of width `4` instead. | For details about using the TimeKey and Clustering Key functionality, see [TimeKeys and Clustering Keys](/timekeys-and-clustering-keys). For details about other index types, see [Secondary Indexes](/secondary-indexes). ## Create a Table for Geospatial Data This example assumes you plan to ingest geospatial data linked to time series data using . The most important columns that you plan to reference frequently in queries include: * `route_time` — A granular timestamp column that represents the primary time reference in the table. * `start_point`, `end_point` — The geospatial point values for the start and destination locations. * `trip_id` — An integer column to identify each unique trip. * `vehicle_id` — An integer column that represents each vehicle. ```sql SQL theme={null} CREATE TABLE "trip_details" ( "route_time" TIMESTAMP TIME KEY BUCKET(14, DAY) NOT NULL, "start_point" POINT NULL, "end_point" POINT NOT NULL DEFAULT 'POINT(0 0)', "route_path" LINESTRING NOT NULL DEFAULT 'LINESTRING(0 0, 1 1)', "route_area" POLYGON NOT NULL DEFAULT 'POLYGON((0 0, 0 1, 1 1, 1 0, 0 0))', "trip_id" BIGINT NOT NULL, "vehicle_id" BIGINT NOT NULL, CLUSTERING KEY "ck" ("trip_id", "vehicle_id") ); CREATE INDEX "start_idx" on "trip_details" ("start_point") USING SPATIAL; CREATE INDEX "end_idx" on "trip_details" ("end_point") USING SPATIAL; ``` This example table makes use of these Ocient System configurations. | **Column Name** | **Configuration Description** | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `route_time` | This column is the table TimeKey, which means that the database indexes for filter optimization.
The example configures the TimeKey to partition segments by every two weeks `(14, DAY)`, which is the approximate time granularity that you expect to use in your query filters. | | `trip_id`, `vehicle_id` | The example table identifies these two columns as the table Clustering Key. The database pairs these two columns for quick reference, such as filtering one column for row values of the other. | | `end_point`, `route_path`, `route_area` | These columns have `POINT`, `LINESTRING`, and `POLYGON` data types, respectively.
Each of these geospatial columns includes default values in its respective format. | | `start_point`, `end_point` | The `CREATE INDEX` SQL statements for both of these columns assign `SPATIAL` indexes for quicker reference. This index is the default type for any column with a geospatial data type. | For details about OcientGeo functionality, see [Geospatial Functions](/geospatial-functions). ## Create a Table With Compression Options This example table includes multiple compression options to ensure minimal storage for specific columns and extra performance. Most data types, except arrays, have `COMPRESSION DYNAMIC` applied by default. ```sql SQL theme={null} CREATE TABLE "call_records" ( "utc_timestamp" TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, "sectorid" VARCHAR COMPRESSION ZSTD, "address" VARCHAR COMPRESSION NONE, "call_type" VARCHAR COMPRESSION ZSTD COMPRESSION GDC(4), "access_information" VARCHAR(500) COMPRESSION GDC(4), "file_name" VARCHAR COMPRESSION ZSTD compression_level = 3, dictionary_size = 1048576, CLUSTERING KEY "ck" ("call_type", "access_information") ); ``` This example table makes use of these Ocient System configurations. | **Column name** | **Configuration Description** | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `address` | This column explicitly removes compression by specifying `COMPRESSION NONE` in the `CREATE TABLE` SQL statement. | | `call_type`, `access_information` | These columns have `VARCHAR` data types configured with Global Dictionary Compression (GDC).
This compression scheme maps the variable-length type rows to integers. Among other benefits, this compression allows both columns to support a Clustering Key for faster querying.
The `call_type` column stacks multiple compression schemes by including both GDC and ZSTD compression to provide additional storage reduction. | | `file_name` | This column uses ZSTD compression with extra options. The `compression_level` is set for `3`, meaning the column data is slightly more compressed than the default `0`. The dictionary\_size is set for the maximum value, `1048576`, which means greater compression but higher memory demand. | For details about Ocient compression schemes, see [Table Compression Options](/table-compression-options). ## Create a Table for Various Container Types This example table includes complex container data types, including a two-dimensional array, a tuple with variable-length values, and a tuple with various data types. ```sql SQL theme={null} CREATE TABLE "container_data" ( "utc_timestamp" TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, "user_id" BIGINT NOT NULL DEFAULT 0, "sell_id" BIGINT NOT NULL DEFAULT 0, "point_array_col" POINT[], "array_col" VARCHAR[] COMPRESSION GDC(4), "2D_array_col" INT[][], "tuple_col1" TUPLE<< VARCHAR(255), VARCHAR(500) >>, "tuple_col2" TUPLE<< INT, INT[], VARCHAR COMPRESSION GDC(1) COMPRESSION ZSTD, VARCHAR >>, "matrix_col" MATRIX[5][10], CLUSTERING INDEX "ck" ("user_id", "sell_id") ); CREATE INDEX "point_idx" on "container_data" ("point_array_col") USING SPATIAL; CREATE INDEX "tuple_idx" on "container_data" ("tuple_col1"[1]) USING HASH; CREATE INDEX "tuple_idx2" on "container_data" ("tuple_col2"[2]) USING INVERTED; ``` This example table makes use of these Ocient System configurations. | **Column name** | **Configuration Description** | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `point_array_col` | This column is an array of `POINT` values that is indexed using a `SPATIAL` index type as specified by the `CREATE INDEX` SQL statement. | | `tuple_col1` | This tuple column contains two variable-length values.
Each object in the tuple can support its own compression scheme and index. The `CREATE INDEX` SQL statement assigns a `HASH` index to the first value (`VARCHAR(255)`). | | `tuple_col2` | This tuple column contains various data types. Each object in the tuple can support its own compression scheme and index.
The two `VARCHAR` objects in the tuple column each have their own unique compression scheme.
The `CREATE INDEX` SQL statement assigns an `INVERTED` index to the second tuple object, the integer array `INT[]`. | For details about Ocient container types, see [Array, Tuple, and Matrix Overview](/array-tuple-and-matrix-overview). ## Create a Table With a Retention Policy This example creates a retention policy that keeps rows only if they are less than one day old, as specified by the last line, `RETENTION POLICY AGE 1 DAY`. The retention policy depends on the `created_at` column because it is the TimeKey column. The TimeKey bucket value `(1, HOUR)` determines how frequently the system checks for any rows to remove. ```sql SQL theme={null} theme={null} CREATE TABLE IF NOT EXISTS "transact_data" ( "created_at" TIMESTAMP TIME KEY BUCKET(1, HOUR) NOT NULL, "user_id" INT NOT NULL DEFAULT 0, "sell_id" BIGINT NOT NULL, "purchase_amount" BIGINT NOT NULL DEFAULT 0, "buyer_name" VARCHAR(1048576), CLUSTERING KEY "primary_index" ("sell_id", "user_id") ) RETENTION POLICY AGE 1 DAY; ``` For details about configuring table retention, see [Table Retention Policies](/table-retention-policies). ## Related Links [Data Types](/data-types) [Data Definition Language (DDL) Statement Reference](/data-definition-language-ddl-statement-reference) [Table Compression Options](/table-compression-options) [Secondary Indexes](/secondary-indexes) # Data Control Language (DCL) Statement Reference Source: https://docs.ocient.com/data-control-language-dcl-statement-reference Reference for Ocient Data Control Language (DCL) statements, including GRANT and REVOKE for managing privileges, roles, and access in the database. In order to assign privileges to users, administrators can use Data Control Language (DCL) statements. You can write and execute DCL statements like any other SQL statements against the System and databases. DCL statements allow you to quickly and easily grant or revoke privileges from users. DCL statements work off two main concepts: Grants and Revokes. Statements that begin with `GRANT` give the associated privileges to a user or group. Statements that begin with `REVOKE` remove those privileges from a user or group. A full list of privileges can be found in the [Ocient Privileges Reference](#ocient-privileges-reference). Supported DCL SQL statements are: * GRANT PRIVILEGE * REVOKE PRIVILEGE * GRANT ROLE * REVOKE ROLE When you create an object in the system, database, or schema, the Ocient System automatically grants you all valid privileges for that type of object. For example, if you create a table in a database, the system grants you all privileges that apply to the created table. For details, see [Table Privileges](#table-privileges). For non-SSO users, the system assigns creator privileges to the user. For SSO users, the system follows specific criteria to assign the privileges. To understand the criteria the system uses, see [User Access Control and Workload Management with SSO](/authentication-methods#user-access-control-and-workload-management-with-sso). ## Grant Privileges The `GRANT` statement grants privileges to a user or group. The privileges that you can grant in the `GRANT` statement map to the tables and differ per object. The `WITH GRANT OPTION` keywords specify that the grantee can also grant privileges on that object. To grant privileges, you must have: * `VIEW` privileges for the specified user or group being granted new privileges. * `SYSAUTH` privileges over the system or database object that the specified user or group is being granted privileges on. **Syntax** ```sql SQL theme={null} GRANT { { CREATE { } | VIEW { SCHEMA | TABLE | VIEW | QUERIES | REDACTED QUERIES } | ALTER { SCHEMA | TABLE | VIEW } | DROP { SCHEMA | TABLE | VIEW } | CANCEL QUERIES | SELECT | DELETE | SYSAUTH | USE | ALL } ON SYSTEM | { CREATE { } | VIEW { SCHEMA | TABLE | VIEW | QUERIES | REDACTED QUERIES } | ALTER { SCHEMA | TABLE | VIEW } | DROP { SCHEMA | TABLE | VIEW } | CANCEL QUERIES | SELECT | ALTER | DROP | DELETE | SYSAUTH | USE | ALL } ON DATABASE object_name | { CREATE { | MLMODEL } | VIEW { } | ALTER { } | DROP { } | VIEW | SELECT | ALTER | DROP | DELETE | SYSAUTH | INSERT | ALL } ON SCHEMA object_name | { VIEW | ALTER | DROP | SYSAUTH | ALL } ON { GROUP | USER } object_name | { SELECT | DROP | SYSAUTH | ALL } ON MLMODEL object_name | { VIEW | EXECUTE | ALTER | DROP | SYSAUTH | ALL } ON PIPELINE object_name | { VIEW | DROP | SYSAUTH | ALL } ON PIPELINE FUNCTION object_name | { VIEW | SELECT | ALTER | DROP | SYSAUTH | ALL } ON VIEW object_name | { VIEW | SELECT | ALTER | LOAD | DROP | DELETE | SYSAUTH | INSERT | ALL } ON TABLE object_name | { VIEW | SELECT | ALL } ON TABLE sys.system_catalog_table } TO { { GROUP | USER } object_name | PUBLIC } [ WITH GRANT OPTION ] ::= DATABASE | TABLE | VIEW | USER | GROUP ::= SCHEMA | TABLE | VIEW | MLMODEL | PIPELINE | PIPELINE FUNCTION | USER | GROUP ::= TABLE | VIEW ``` When you grant any privilege to a user on a system or database object other than the `VIEW` privilege, the database implicitly grants the `VIEW` privilege on the specified object to the user, as well as associated object types (e.g., `DROP VIEW ON DATABASE` also grants `VIEW VIEW ON DATABASE` and `VIEW ON DATABASE`). **Examples** **Grant Privilege on a Table** This example grants privileges for the `SELECT` SQL statement on a table to a trusted group. ```sql SQL theme={null} GRANT SELECT ON TABLE company_data TO GROUP trusted_employees; ``` **Grant Privilege on a Database** This example grants the `SELECT` privilege for all tables and views on the `database_name` database to a trusted group. ```sql SQL theme={null} GRANT SELECT ON DATABASE database_name TO GROUP trusted_employees; ``` **Grant Privilege on the System** This example grants the `SELECT` privilege for all tables and views on all databases in the system to a trusted group. ```sql SQL theme={null} GRANT SELECT ON SYSTEM TO GROUP trusted_employees; ``` For examples of granting privileges for individual users, see [Grant Role Membership](#grant-role-membership). ## Revoke Privileges The `REVOKE` statement revokes privileges from a user or group. To revoke privileges, you must have: * `VIEW` privileges for the specified user or group having their privileges revoked. * `SYSAUTH` privileges over the system or database object from which the specified user or group has their privileges revoked. ```sql SQL theme={null} REVOKE [ GRANT OPTION FOR ] { { CREATE { } | VIEW { SCHEMA | TABLE | VIEW | QUERIES | REDACTED QUERIES } | ALTER { SCHEMA | TABLE | VIEW } | DROP { SCHEMA | TABLE | VIEW } | CANCEL QUERIES | SELECT | DELETE | SYSAUTH | USE | ALL } ON SYSTEM | { CREATE { } | VIEW { SCHEMA | TABLE | VIEW | QUERIES | REDACTED QUERIES } | ALTER { SCHEMA | TABLE | VIEW } | DROP { SCHEMA | TABLE | VIEW } | CANCEL QUERIES | SELECT | ALTER | DROP | DELETE | SYSAUTH | USE | ALL } ON DATABASE object_name | { CREATE { | MLMODEL } | VIEW { } | ALTER { } | DROP { } | VIEW | SELECT | ALTER | DROP | DELETE | SYSAUTH | INSERT | ALL } ON SCHEMA object_name | { VIEW | ALTER | DROP | SYSAUTH | ALL } ON { GROUP | USER } object_name | { SELECT | DROP | SYSAUTH | ALL } ON MLMODEL object_name | { VIEW | EXECUTE | ALTER | DROP | SYSAUTH | ALL } ON PIPELINE object_name | { VIEW | DROP | SYSAUTH | ALL } ON PIPELINE FUNCTION object_name | { VIEW | SELECT | ALTER | DROP | SYSAUTH | ALL } ON VIEW object_name | { VIEW | SELECT | ALTER | LOAD | DROP | DELETE | SYSAUTH | INSERT | ALL } ON TABLE object_name | { VIEW | SELECT | ALL } ON TABLE sys.system_catalog_table } FROM { [ USER | GROUP ] object_name | PUBLIC } ::= DATABASE | TABLE | VIEW | USER | GROUP ::= SCHEMA | TABLE | VIEW | MLMODEL | PIPELINE | PIPELINE FUNCTION | USER | GROUP ::= TABLE | VIEW ``` When you revoke the `VIEW` privilege on a system or database object, the database implicitly revokes all privileges on the specified object from the user. If you revoke the `VIEW` privilege on an associated object type, the system revokes the privileges for that object type. For example, grant the `DROP TABLE` privilege to the user by using the `GRANT DROP TABLE ON DATABASE` SQL statement. Due to implicit granting by the system, this action causes the user to have `DROP TABLE`, `VIEW TABLE`, and `VIEW` privileges. If you revoke the `VIEW TABLE` privilege using the `REVOKE VIEW TABLE ON DATABASE` SQL statement, then the system revokes the privileges for the associated type. In this case, the user only retains the `VIEW` privilege on the database. **Example** This example revokes the SELECT privilege on a table from an untrusted group. ```sql SQL theme={null} REVOKE SELECT ON TABLE company_data FROM GROUP untrusted; ``` For examples of revoking privileges for individual users, see [Revoking Role Membership](#revoke-role-membership). ### Ocient Privileges Reference These tables describe the privilege options on each object in Ocient and the allowed privileges. As of version 23.0, Ocient DCL has replaced the `TRUNCATE` privilege with the `DELETE` privilege. Any `TRUNCATE` privileges, whether manually granted or assigned as part of a user role, should automatically convert to `DELETE` privileges following a system upgrade to version 23.0 or later. #### Ocient System Privileges | **Privilege** | **Description** | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CREATE \[DATABASE, USER, GROUP] | The privilege to create the specified object type. On creation, the creating user inherits all privileges on the object. In the case of the DATABASE object, this is enforced by granting the creating user the newly created database administrator role as described in the Roles section. | | CREATE \[SCHEMA, TABLE, VIEW] | The privilege to create the specified object type on any database where the user can connect. On creation, the creating user inherits all privileges on the object. In order to apply the privilege, the user has to be connected to a user-defined database. | | DROP \[SCHEMA, TABLE, VIEW] | The privilege to drop the specified object type on any database where the user can connect. In order to apply the privilege, the user has to be connected to a user-defined database. | | ALTER \[SCHEMA, TABLE, VIEW] | The privilege to modify the specified object type on any database where the user can connect. In order to apply the privilege, the user has to be connected to a user-defined database. | | VIEW \[SCHEMA, TABLE, VIEW] | The privilege to see and read information of the specified object type within the system. | | VIEW QUERIES | The privilege to see the full SQL statement and metadata for all queries within the system. | | VIEW REDACTED QUERIES | The privilege to see the metadata for all queries with a fully redacted SQL statement within the system. | | CANCEL QUERIES | The privilege to cancel or kill running queries within the system. | | SELECT | The privilege to read data on any table or view where the user has the VIEW privilege. | | SYSAUTH | The privilege to grant and revoke all privileges on the system. | | USE | The ability to connect to the system. | | DELETE | The privilege to delete data from and truncate tables within the system. | #### Database Privileges | **Privilege** | **Description** | | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | CREATE \[SCHEMA, TABLE, VIEW, MLMODEL, PIPELINE, PIPELINE FUNCTION, USER, GROUP] | The privilege to create the specified object type. The user who creates the object inherits all privileges on the object. | | DROP \[SCHEMA, TABLE, VIEW] | The privilege to drop the specified object type on this database. | | ALTER \[SCHEMA, TABLE, VIEW] | The privilege to modify the specified object type on this database. | | VIEW \[SCHEMA, TABLE, VIEW] | The privilege to see and read information of the specified object type within the database. | | VIEW QUERIES | The privilege to see the full SQL statement and metadata for all queries in this database. | | VIEW REDACTED QUERIES | The privilege to see the metadata for all queries with a fully redacted SQL statement in this database. | | CANCEL QUERIES | The privilege to cancel or kill running queries in this database. | | SELECT | The privilege to read data on any table or view, where the user has the VIEW privilege, in the database. | | USE | Ability to connect to the database. Users created in this database are implicitly granted this privilege. | | ALTER | The ability to issue ALTER DATABASE statements on this database. | | DROP | The ability to drop this database. | | SYSAUTH | The ability to grant and revoke all privileges on the database. | | DELETE | The privilege to delete data from and truncate tables in the database. | #### Schema Privileges | **Privilege** | **Description** | | ------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | CREATE \[ TABLE \| VIEW \| MLMODEL ] | The privilege to create the specified object type. | | DROP \[ TABLE \| VIEW ] | The privilege to drop the specified object type within the schema. | | ALTER \[ TABLE \| VIEW ] | The privilege to modify the specified object type within the schema. | | VIEW \[ TABLE \| VIEW ] | The privilege to see and read information of the specified object type within the schema. | | SELECT | The privilege to read data on any table or view, where the user has the VIEW privilege, within the schema. | | ALTER | The ability to modify the schema. | | DROP | The privilege to drop the schema and its contents. | | VIEW | The ability to see and read information about the schema. | | SYSAUTH | The ability to grant and revoke all privileges on the schema. | | INSERT | The privilege to insert data into tables in the schema. | | DELETE | The privilege to delete data from and truncate tables in the schema. | Granting privileges using DCL statements promotes an implicit schema to an explicit schema. For details, see [Schemas](/schemas). #### View Privileges | **Privilege** | **Description** | | ------------- | ---------------------------------------------------------- | | VIEW | Ability to see and read schema information about the view. | | SELECT | Ability to read data from the view. | | ALTER | Ability to modify the view. | | DROP | Ability to drop the view. | | SYSAUTH | Ability to grant privileges on the view. | #### Table Privileges | **Privilege** | **Description** | | ------------- | ----------------------------------------------------------- | | VIEW | Ability to see and read schema information about the table. | | SELECT | Ability to read data from the table. | | ALTER | Ability to modify the table. | | LOAD | Ability to load data into the table. | | DROP | Ability to drop the table. | | DELETE | Ability to delete and truncate the table. | | SYSAUTH | Ability to grant and revoke privileges on the table. | | INSERT | Ability to insert data into the table. | #### Machine Learning Model Privileges | **Privilege** | **Description** | | ------------- | --------------------------------------------------------- | | ALTER | Ability to rename the model. | | SELECT | Ability to read data and descriptive data from the model. | | DROP | Ability to drop the model. | | SYSAUTH | Ability to grant and revoke privileges on the model. | #### Data Pipeline Privileges | **Privilege** | **Description** | | ------------- | ------------------------------------------------- | | VIEW | See and read information schema about a pipeline. | | EXECUTE | Start and stop the execution of a pipeline. | | ALTER | Rename and replace a pipeline. | | DROP | Drop a pipeline. | | SYSAUTH | Grant and revoke privileges on a pipeline. | #### Data Pipeline Function Privileges | **Privilege** | **Description** | | ------------- | -------------------------------------------------------------------------------------------- | | VIEW | See and read information schema about a data pipeline function in the system catalog tables. | | DROP | Drop a data pipeline function. | | SYSAUTH | Grant and revoke privileges on a data pipeline function. | #### User Privileges | **Privilege** | **Description** | | ------------- | ---------------------------------------------------------- | | VIEW | Ability to read information about the user. | | ALTER | Ability to modify the user. | | DROP | Ability to drop the user. | | SYSAUTH | Ability to grant rights and revoke privileges on the user. | #### Group Privileges | **Privilege** | **Description** | | ------------- | -------------------------------------------- | | VIEW | Ability to read information about the group. | | ALTER | Ability to alter the group. | | DROP | Ability to drop the group. | | SYSAUTH | Ability to grant privileges on the group. | #### System Catalog and Object or View Visibility The `sys.privileges` table in the system catalog exposes the privileges in the system. This table displays this information: * Timestamp of the grant * Grantor * Grantee * Privilege granted * Object type * Object id * Grantable You can grant and revoke System Catalog `VIEW` and `SELECT` privileges just like ordinary tables. By default, everyone has `VIEW` and `SELECT` privileges on all system catalog tables. You can see the objects within the system catalog tables only if you have sufficient privileges on those objects. These objects require a `VIEW` or `SELECT` privilege or membership in a group or role to provide visibility. When you execute queries against a system catalog table, you do not see or know the existence of objects to which you do not have access. Views offer a similar functionality because the database does not check privileges to the underlying tables and views after you create a view. You can create a table with sensitive information and restrict visibility by creating a view on top of the table with only certain rows or columns. Someone else with the `SELECT` privilege to the view can query the view, even without any privileges to the underlying table. ## Roles Similar to groups, users can inherit privileges by being granted one of the predefined roles in Ocient. The names of and privileges assigned to roles in Ocient are predefined by the system. Roles can be applicable to the Ocient System or one of the user-defined databases. The roles in Ocient are as follows: **System Roles** * **Security Administrator** — Can read, create, and modify all users and groups. * **System Administrator** — Can read, create, and modify system objects such as tables, clusters, databases, etc. Can see and delete any queries system-wide. * **System Analyst** — Read-only access to the entire system. Can see all queries in the system. **Database Roles** * **Database Administrator** — Can read, create, and modify database objects such as tables, clusters, databases, etc. This role can also create users for the database. Can see and delete all queries in the database. * **Database Analyst** — Read-only rights to database objects and data within the database. Can see all queries in the database. * **Public** — Every user who has access to the database has the Public role by default. You can grant other privileges to this role. This role allows the creation of a schema using the `CREATE SCHEMA` SQL statement and allows the viewing of other roles by default. For specifics on each Ocient role, see [Default Role Privileges](#default-role-privileges). ### Grant Role Membership Grants a role to a user or group. **Syntax** ```sql SQL theme={null} GRANT ROLE role_name TO { USER | GROUP } user_or_group_name ``` | **Parameter** | **Data** **Type** | **Description** | | -------------------- | ----------------- | ----------------------------------------------------------------------------------------------------- | | `role_name` | string | The name of a role to be granted to the specified user or group.
Enclose this string in quotes. | | `user_or_group_name` | string | The name of a user or group to be granted the specified role. | **Example** This example grants `user1` the system administrator role. ```sql SQL theme={null} GRANT ROLE "System Administrator" TO USER user1; ``` ### Revoke Role Membership Revokes a role from a user or group. **Syntax** ```sql SQL theme={null} REVOKE ROLE role_name FROM { USER | GROUP } user_or_group_name ``` | **Parameter** | **Data** **Type** | **Description** | | -------------------- | ----------------- | ------------------------------------------------------------------------------------------------------ | | `role_name` | string | The name of a role to be revoked for the specified user or group.
Enclose this string in quotes. | | `user_or_group_name` | string | The name of a user or group to have the specified role revoked. | **Example** This example revokes the system administrator role from `user1`. ```sql SQL theme={null} REVOKE ROLE "System Administrator" FROM USER user1; ``` ### Default Role Privileges #### Security Administrator Privileges  | **Target** | **Privileges** | | ---------------------- | -------------------------------- | | System | SYSAUTH, CREATE \[ USER ] | | Schema | SYSAUTH | | Database | SYSAUTH, CREATE \[ USER, GROUP ] | | View | SYSAUTH | | Table | SYSAUTH | | Machine Learning Model | SYSAUTH | | Data Pipeline | SYSAUTH | | Data Pipeline Function | SYSAUTH | | User | VIEW, ALTER, DROP, SYSAUTH | | Group | VIEW, ALTER, DROP, SYSAUTH | #### System Administrator Privileges  | **Target** | **Privileges** | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------ | | System | USE, CREATE \[ DATABASE, USER ], VIEW QUERIES, CANCEL QUERIES | | Schema | VIEW, ALTER, DROP | | Database | - USE
- ALTER
- DROP
- CREATE \[ SCHEMA, TABLE, VIEW, MLMODEL, USER, GROUP, PIPELINE, PIPELINE FUNCTION ] | | View | VIEW, SELECT, ALTER, DROP | | Table | VIEW, SELECT, ALTER, LOAD, DROP, DELETE, INSERT | | Machine Learning Model | SELECT, DROP | | Data Pipeline | VIEW, EXECUTE, ALTER, DROP | | Data Pipeline Function | VIEW, DROP | | User | VIEW, ALTER, DROP | | Group | VIEW, ALTER, DROP | #### System Analyst Privileges  | **Target** | **Privileges** | | ---------------------- | ----------------- | | System | USE, VIEW QUERIES | | Schema | VIEW | | Database | USE | | View | VIEW, SELECT | | Table | VIEW, SELECT | | Machine Learning Model | SELECT | | Data Pipeline | VIEW | | Data Pipeline Function | VIEW | | User | VIEW | | Group | VIEW | #### Database Administrator Privileges  | **Target** | **Privileges** | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Schema | VIEW, ALTER, DROP, SYSAUTH | | Database | - USE
- ALTER
- DROP
- CREATE \[ SCHEMA, TABLE, VIEW, MLMODEL, USER, GROUP, PIPELINE, PIPELINE FUNCTION ]
- VIEW QUERIES, CANCEL QUERIES
- SYSAUTH | | View | VIEW, SELECT, ALTER, DROP, SYSAUTH | | Table | VIEW, SELECT, ALTER, LOAD, DROP, DELETE, INSERT, SYSAUTH | | Machine Learning Model | SELECT, DROP, SYSAUTH | | Data Pipeline | VIEW, EXECUTE, ALTER, DROP, SYSAUTH | | Data Pipeline Function | VIEW, DROP, SYSAUTH | | User | VIEW, ALTER, DROP, SYSAUTH | | Group | VIEW, ALTER, DROP, SYSAUTH | #### Database Analyst Privileges  | **Target** | **Privileges** | | ---------------------- | ----------------- | | Schema | VIEW | | Database | USE, VIEW QUERIES | | View | VIEW, SELECT | | Table | VIEW, SELECT | | Machine Learning Model | SELECT | | Data Pipeline | VIEW | | Data Pipeline Function | VIEW | | User | VIEW | | Group | VIEW | #### Public Privileges  | **Target** | **Privileges** | | ---------- | -------------- | | Database | CREATE SCHEMA | ### Query Visibility The visibility of queries depends on which database you are logged into and your assigned privileges. Users with no additional VIEW QUERIES or VIEW REDACTED QUERIES privileges can view only queries that they submit themselves. The VIEW QUERIES privilege allows access to queries for all users in these system catalog tables: * `sys.queries ` * `sys.completed_queries` * `sys.plans ` * `sys.active_query_scheduler_info ` * `sys.all_operator_instances ` * `sys.completed_operator_instances ` * `sys.op_inst_debug_info` The VIEW REDACTED QUERIES privilege allows access to queries for all users in these system catalog tables, but with a redacted SQL statement for queries from other users: * `sys.queries ` * `sys.completed_queries` For the initial grant of query privileges, the grantor must have the `Database Administrator` or `System Administrator` role. Use the `WITH GRANT OPTION` keywords to allow a user to further grant this privilege to other users and groups. #### Login Impacts on Query Visibility Along with having role privileges, users also must be logged into the appropriate database to view queries: * If the user logs in to the `SYSTEM` database, the user can see all queries from all databases. * If the user logs in to a database other than `SYSTEM`, the user can see queries in only that database. ## Related Links [Database Administration](/database-administration) [Manage Users, Groups, and Roles](/manage-users-groups-and-roles) [Object-Type Level Privileges Management](/object-type-level-privileges-management) # Data Definition Language (DDL) Statement Reference Source: https://docs.ocient.com/data-definition-language-ddl-statement-reference Reference for Ocient Data Definition Language (DDL) statements, including CREATE, ALTER, and DROP for databases, tables, views, indexes, and other objects. DDL statements allow users to run commands and administrative operations on the System. You can execute DDL statements over a database connection to manage nodes, storage spaces, tables and views, system configuration, users and groups, underlying storage segments, and more. **Quoting Identifiers** In all DDL statements, identifiers not in double quotes must begin with a letter and can only contain letters, numbers, and underscores. Identifiers are the names of databases, tables, nodes, and so on. For details, see [Identifiers](/identifiers). The Ocient System internally converts identifiers to lower-case. Identifiers in double quotes can contain any characters besides newline and carriage return, can begin with any character, and are not case-adjusted. To use an identifier with the same name as any keyword, it must be in double quotes. String literals must be enclosed in single quotes. A single quote within a string can be escaped as "\*. \*A string literal can be preceded by e to enable additional escape sequences (for example: `e'\n'`). DDL statements are supported for these categories: ## [Cluster and Node Management](/cluster-and-node-management) * **CLUSTER** * CREATE CLUSTER * DROP CLUSTER * ALTER CLUSTER * ALTER CLUSTER ADD PARTICIPANTS * ALTER CLUSTER DROP PARTICIPANTS * ALTER CLUSTER ADD STORAGESPACE * ALTER CLUSTER REMOVE STORAGESPACE * ALTER CLUSTER ALTER CONFIG SET * ALTER CLUSTER ALTER LOG LEVEL SET * ALTER CLUSTER RENAME * **STORAGESPACE** * CREATE STORAGESPACE * DROP STORAGESPACE * ALTER STORAGESPACE RENAME * **NODE** * DROP NODE * ALTER NODE * ALTER NODE ADD ROLE * ALTER NODE REMOVE ROLE * ALTER NODE ALTER CONFIG SET * ALTER NODE RENAME * ALTER NODE ALTER LOG LEVEL SET * ALTER NODE ALTER METRIC LEVEL * ALTER NODE SET ADDRESS * **CONNECTIVITY POOL** * CREATE CONNECTIVITY\_POOL * DROP CONNECTIVITY\_POOL * ALTER CONNECTIVITY\_POOL * ALTER CONNECTIVITY\_POOL SET * ALTER CONNECTIVITY\_POOL RENAME TO * ALTER CONNECTIVITY\_POOL ADD PARTICIPANTS * ALTER CONNECTIVITY\_POOL ALTER PARTICIPANT * ALTER CONNECTIVITY\_POOL DROP PARTICIPANTS * ALTER CONNECTIVITY\_POOL SET SSO INTEGRATION * ALTER CONNECTIVITY\_POOL REMOVE SSO INTEGRATION * SSO INTEGRATION * CREATE SSO INTEGRATION * DROP SSO INTEGRATION * ALTER SSO INTEGRATION * **SYSTEM** * ALTER SYSTEM ALTER CONFIG SET * ALTER SYSTEM RENAME TO * ALTER SYSTEM ALTER METRIC LEVEL * ALTER SYSTEM ALTER SECURITY * ALTER SYSTEM SET DEFAULT STORAGESPACE ## [Schemas](/schemas) * CREATE SCHEMA * DROP SCHEMA * ALTER SCHEMA RENAME ## [Databases](/databases) * CREATE DATABASE * DROP DATABASE * ALTER DATABASE * ALTER DATABASE RENAME * ALTER DATABASE SET SSO INTEGRATION * ALTER DATABASE ALTER SSO INTEGRATION * ALTER DATABASE REMOVE SSO INTEGRATION * ALTER DATABASE ALTER SECURITY ## [Tables](/tables) * CREATE TABLE * CREATE TABLE AS SELECT (CTAS) * CREATE TABLE AS SELECT USING LOADERS * DROP TABLE * ALTER TABLE * ALTER TABLE RENAME * ALTER TABLE RENAME COLUMN * ALTER TABLE ADD COLUMN * ALTER TABLE ALTER COLUMN COMPRESSION * ALTER TABLE ALTER REDUNDANCY * ALTER TABLE DROP COLUMN * ALTER TABLE STREAMLOADER\_PROPERTIES * ALTER TABLE DISABLE INDEX * ALTER TABLE ENABLE INDEX * ALTER TABLE ENABLE RETENTION POLICY AGE * ALTER TABLE DISABLE RETENTION POLICY * DELETE FROM TABLE * EXPORT TABLE * INSERT INTO TABLE * INSERT INTO TABLE USING LOADERS * TRUNCATE TABLE ## [Views](/views) * CREATE VIEW * DROP VIEW * ALTER VIEW RENAME * ALTER VIEW AS * EXPORT VIEW ## [Indexes](/indexes) * CREATE INDEX * DROP INDEX ## [Data Pipelines](/data-pipelines) * PIPELINE * CREATE PIPELINE * DROP PIPELINE * PREVIEW PIPELINE * START PIPELINE * STOP PIPELINE * ALTER PIPELINE * ALTER PIPELINE RENAME * EXPORT PIPELINE * CREATE PIPELINE FUNCTION * DROP PIPELINE FUNCTION ## [Distributed Tasks](/distributed-tasks) * **TASK** * CREATE TASK * CANCEL TASK ## [Machine Learning Models](/machine-learning-models) * CREATE MLMODEL * ALTER MLMODEL * DROP MLMODEL * EXPORT MLMODEL * REFRESH MLMODEL ## [Users, Groups, and Service Classes](/users-groups-and-service-classes) * **USER** * CREATE USER * DROP USER * ALTER USER * ALTER USER SET * **GROUP** * CREATE GROUP * DROP GROUP * ALTER GROUP * ALTER GROUP USER * ALTER GROUP RENAME * ALTER GROUP SET SERVICE CLASS * ALTER GROUP ALTER SECURITY * **SERVICE CLASS** * CREATE SERVICE CLASS * DROP SERVICE CLASS * ALTER SERVICE CLASS * ALTER SERVICE CLASS RENAME * ALTER SERVICE CLASS SET * ALTER SERVICE CLASS RESET * ALTER QUERY ## [Data Integrity and Storage](/data-integrity-and-storage) * DRAIN PAGES * ALTER SEGMENT QUARANTINE ## [Query Analysis](/query-analysis) * EXPLAIN * EXPLAIN PIPELINE ## [Query Management](/query-management) * CANCEL * KILL ## [Statistics Cache Management](/statistics-cache-management) # Data Extract Tool Source: https://docs.ocient.com/data-extract-tool Use the Ocient Data Extract tool with the JDBC driver to export rows from tables and queries into files for backup, migration, or downstream processing. The data extract tool is a part of the JDBC driver to unload data. You can execute the tool directly from the JDBC CLI. The tool extracts a result set to delimited or files in the target location. To invoke the JDBC CLI, see the [JDBC Manual](/jdbc-manual). To use the data extract tool, you must have JDBC version 2.63 or higher. ## Supported Data Extract Formats The data extract tool supports unloading result sets into files in specific formats. Supported extract formats are: * CSV — Outputs result sets as text files with fields separated by a chosen delimiter. * Parquet — Outputs result sets as Parquet files. ## General Command Structure Here is the general structure of an extract command. ```sql SQL theme={null} EXTRACT TO [OPTIONS([param=value [,...]])] AS ``` The command is case-insensitive. Each extract command must start with `EXTRACT TO`. The location type `location_type` must be either `LOCAL` for the local machine or `S3` for S3. You can enclose additional options within a pair of parentheses following the keyword `OPTIONS`. The location type is required and there are required options for each location type. For the `LOCAL` location type, the options must define the file prefix `file_prefix`, and for `S3`, the options must define the file prefix `file_prefix`, bucket `bucket`, and endpoint `endpoint`. Next, the query follows the keyword `AS`. This example is a simple general command structure. ```sql SQL theme={null} EXTRACT TO LOCAL OPTIONS( file_prefix="/home/user/out/data_", file_extension=".csv" ) AS SELECT c1 FROM sys.dummy10; ``` For supported options, see [Data Extract Options](#data-extract-options). ## Specify Options, Quoting, and Escaping Quotes Here is the general format of options. ```sql SQL theme={null} key1 = value1, key2 = value2, ... , keyN = valueN ``` You need to follow certain guidelines when you specify options. Keys (option names) can only consist of alphanumeric characters and are unquoted. Values can be either quoted (with the reserved character `"`) or unquoted. If values are unquoted, they can only contain alphanumeric characters. If the value has a non-alphanumeric character, you must quote it with the reserved character `"`. Note that the single quote character does not work. ```sql SQL theme={null} OPTIONS(file_prefix = "/path/to/dir/result", header_mode = none, file_extension = ".csv") ``` To use the reserved quote character `"` as an argument, you must escape it with the backslash character `\`. To use `\` as an argument, you must escape it with another `\`. This code illustrates both of these scenarios. ```sql SQL theme={null} OPTIONS(field_optionally_enclosed_by = "\"", escape = "\\") ``` ## Data Extract Options Use these optional options with the `EXTRACT TO` syntax and the `OPTIONS` keyword to configure the behavior of the extract. ### General Extract Options This table describes optional options that apply to both the `LOCAL` and `S3` location types. | **Option** | **Description** | **Default** | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `FILE_TYPE` | The type of the output file for extraction. Supports extraction to a delimited `.csv` file or Parquet `.parquet` file. | `DELIMITED` | | `FILE_PREFIX` | Dictates the prefix used on the results. When extracting to `LOCAL`, this is the prefix used to determine the path of the results. This value can be a relative or full path. When extracting to S3, this value is the prefix for the key. In either case, the system adds additional file numbers and file extensions to generate the complete filename. | `results-` | | `FILE_PREFIX_EXISTS` | Determines the behavior if the path specified by the `FILE_PREFIX` option already exists. Supported values are: `FAIL` and `OVERWRITE`. The `FAIL` value throws an error, whereas `OVERWRITE` deletes the contents of the path. | `'FAIL'` | | `FILE_EXTENSION` | The file extension specified for each output file. | `.csv` | | `MAX_ROWS_PER_FILE` | If you set this option to a non-zero value, the system splits the results into files with the specified maximum number of rows per file. | `NULL` | | `COMPRESSION` | Compression type to use for a delimited extract. Supported compression types are:
`NONE` — No compression
`GZIP` — GZip compression
`BZIP2` — bzip2 compression
`XZ` — xz compression | `NONE` | | `RECORD_DELIMITER` | Delimiter to use between records. This supports strings, so special characters can be input using escape characters. UTF-16: `\u[utf-16 value]` or Octal `\[octal value]`. | `\n` | | `FIELD_DELIMITER` | Delimiter to use between fields within a record. This supports Java strings, so special characters can be input using escape characters. UTF-16: `\u[utf-16 value]` or Octal `\[octal value]`. | `,` | | `HEADER_MODE` | Dictates how to manage headers in result files. Supported values are `NONE`, `ALL_FILES`, and `FIRST_FILE`.

`NONE` — The tool writes all output files without an additional header.
`ALL_FILES` — The tool adds column names as a header in the first row of each output file. Each file has at most `MAX_ROWS_PER_FILE` + 1 total rows.
`FIRST_FILE` — The tool adds column names as a header in the first row of the first output file. The tool does not add the header to subsequent files. Each file has at most `MAX_ROWS_PER_FILE` total rows, inclusive of the header in the first file. | `NONE` | | `NULL_FORMAT` | Format string to use for writing NULL values to the output files. | `""` (empty string) | | `ENCODING` | Encoding used when writing out data to files. | The default character set of the system, as determined by the [Oracle documentation](https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html). | | `ESCAPE` | Character used for escaping quoted fields. Set this to the NULL character `\0` to indicate that the escape character is not specified. | `"` | | `FIELD_OPTIONALLY_ENCLOSED_BY` | Sometimes, you need to surround fields in a character. For example, the field might have a literal comma. Generally, this character is also known as the quote character. Set this option to the NULL character `\0` to indicate that the quote character is not specified. | `"` | | `BINARY_FORMAT` | The format with which to encode the BINARY data type. Supports `UTF-8`, `Hexadecimal`, and `Base64`. | `Hexadecimal` | | `COMPRESSION_BLOCK_SIZE` | The number of bytes that comprise each block to be compressed; larger blocks result in better compression at the expense of more RAM usage when compressing. | `4194304` | | `COMPRESSION_LEVEL` | An integer value \[-1, 9]. Use `-1` for the GZip default compression level, `0` for no compression, or a value \[1-9] where 1 indicates fastest compression and 9 indicates best compression. | `1` | | `NUM_COMPRESSION_THREADS` | The number of threads to use for compression. | \$(number of cores \* 2) | | `NUM_FETCH_QUERIES` | The number of parallel queries to execute in the database for data extraction. | `15` | | `ESCAPE_UNQUOTED_VALUES` | Dictates whether to write escape sequences in unquoted values. Only applicable when `FIELD_DELIMITER` is set to `,`. | `false` | | `INPUT_ESCAPED` | Dictates whether the input is already escaped. When this option is set to true, the tool does not add escape sequences, and data is written without changes to the output file. Only applicable when `FIELD_DELIMITER` is set to `,`.
Ensure that data is properly escaped, otherwise the extract might produce invalid CSV data. | `false` | | `PARTITION_MODE` | The strategy for partitioning the data. Supported values are: `NONE`, `KEY`, and `RANGE`.

When you set this option to `NONE`, the tool uses standard extraction. When you set this option to `KEY`, the tool creates subdirectories for each unique value specified by the `PARTITION_COLUMNS` option. When you set this option to `RANGE`, the tool splits the data into the number of queries specified by the `NUM_FETCH_QUERIES` option based on the range of values specified in the `PARTITION_COLUMNS` option. | `NONE` | | `PARTITION_COLUMNS` | The comma-separated list of columns to use for partitioning data when you set the `PARTITION_MODE` option to `KEY` or `RANGE`. The `RANGE` value only allows a single column. See [File Naming Conventions](#file-naming-conventions) for the file path structure for multiple partitioning columns when using the `KEY` value. | `NULL` | | `QUOTE_ALL_FIELDS` | Dictates whether all written fields are enclosed with quotes. When this option is set to true, the tool encloses all fields with the `FIELD_OPTIONALLY_ENCLOSED_BY` character. | `false` | | `SUCCESS_MARKER` | Identifies a successful completion of the extract. If you set this option to `true`, the tool creates a file with the `_SUCCESS` suffix in the root output directory when the extract of the entire job completes successfully. | `true` | | `TARGET_FILE_SIZE_MB` | Specifies the size in megabytes for the target output file. The data extract tool splits the output into files of approximately this size. The tool ignores this option if you set the `MAX_ROWS_PER_FILE` option. | `NULL` | | `TRANSLATE_CHARACTERS_MODE` | Character Mode to use for translating characters. Supported values are `CHAR` and `HEX`.
The tool performs character translation only if you specify `TRANSLATE_CHARACTERS_FROM` and `TRANSLATE_CHARACTERS_TO`. The tool replaces the Nth character in `TRANSLATE_CHARACTERS_FROM` with the Nth character in `TRANSLATE_CHARACTERS_TO` in the extracted records.
When `TRANSLATE_CHARACTERS_MODE` is set to CHAR, `TRANSLATE_CHARACTERS_FROM`, and `TRANSLATE_CHARACTERS_TO` must be equal-length strings of UTF-8 characters. For example:
`TRANSLATE_CHARACTERS_MODE="CHAR"`,
`TRANSLATE_CHARACTERS_FROM="àëï"`,
`TRANSLATE_CHARACTERS_TO="aei"`

When `TRANSLATE_CHARACTERS_MODE` is set to `HEX`, `TRANSLATE_CHARACTERS_FROM`, and `TRANSLATE_CHARACTERS_TO` must be comma-separated lists of hexadecimal UTF-8 code points with the same number of list elements. For example:
`TRANSLATE_CHARACTERS_MODE="HEX"`,
`TRANSLATE_CHARACTERS_FROM="c3a0,c3ab,c3af"`,
`TRANSLATE_CHARACTERS_TO="61,65,69"` | `CHAR` | | `TRANSLATE_CHARACTERS_FROM` | Sequence of UTF-8 characters in the source data to translate to a corresponding character in the `TRANSLATE_CHARACTERS_TO` option. See the `TRANSLATE_CHARACTERS_MODE` option for the expected format. | `""` | | `TRANSLATE_CHARACTERS_TO` | Sequence of UTF-8 characters to use as a replacement for the characters included in `TRANSLATE_CHARACTERS_FROM`. See the `TRANSLATE_CHARACTERS_MODE` option for the expected format. | `""` | | `TRIM_TRAILING_ZEROS` | Dictates whether to trim trailing zeros from numeric input fields. | `false` | | `PARQUET_COMPRESSION` | Compression type to use for a Parquet extract. Supported compression types are:
`NONE` — No compression
`ZSTD` — ZSTD compression
`SNAPPY` — Snappy compression
`GZIP` — GZip compression | `SNAPPY` | | `PARQUET_ROW_GROUP_SIZE_BYTES` | The size in bytes for row groups within a Parquet output file. | `536870912` (512 MB) | If you do not set either the `MAX_ROWS_PER_FILE` or the `TARGET_FILE_SIZE_MB` options, the data extract tool generates one output file for each partition. For a query without partitions, the tool generates a single output file. ### S3 Extract Options This table describes the required options that apply only to the `S3` location type. The data extract tool ignores these options when you use the `LOCAL` location type. | **Option** | **Description** | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `BUCKET` | S3 bucket to use. | | `ENDPOINT` | Endpoint for S3 upload. For details, see the documentation for [specifying endpoints](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/specifying-endpoints.html). | This table describes optional options that apply only to the `S3` location type. | **Option** | **Description** | **Default** | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | `AWS_KEY_ID` | AWS Key ID. If empty, the CLI uses the Java AWS SDK default credentials provider chain documented [here](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). | `""` | | `AWS_SECRET_KEY` | AWS Secret Key. If empty, the CLI uses the Java AWS SDK default credentials provider chain documented [here](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). | `""` | | `REGION` | S3 region to upload to. Ignored when extracting to LOCAL. | `US_EAST_2` | | `PATH_STYLE_ACCESS` | Whether [path style access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html) should be used to access a bucket. | `true` for JDBC version 3.4.1 and `false` for JDBC version 3.4.0 and earlier | ## File Naming Conventions When you use the data extract tool, the tool produces a number of files. The System determines the path of these files using multiple factors: * File prefix as specified by the `FILE_PREFIX` option * If the `PARTITION_MODE` option is not set to `NONE`, the file prefix must be a directory and end with a forward slash. The system creates subdirectories in the style for each partition (for example, for the `KEY` partition: `/tmp/extract/col1=val1/col2=val2` or for the `RANGE` partition: `/tmp/extract/val1<=col1 `sys.dummy` creates a virtual table with the specified number for rows. For details, see [Generate Tables Using sys.dummy](/generate-tables-using-sys-dummy). **Local CSV Extract to the Specified Path** This example extracts the results of the `SELECT c1 FROM sys.dummy10` SQL query to the local machine at the absolute path `/home/user/out/data_0.csv`. The SQL query returns 10 rows with incremental numbers starting at 1. ```sql SQL theme={null} EXTRACT TO LOCAL OPTIONS( file_prefix="/home/user/out/data_", file_extension=".csv" ) AS SELECT c1 FROM sys.dummy10; ``` **Local Parquet Extract with Partition by Key** This example partitions customer data into a Parquet file by the `country` column. This partitioning creates a Hive-style directory structure on the local machine. Use the file path `/home/user/customer_data/` with the `KEY` value for the partition strategy. Specify the target file size to be `256` MB. The SQL query selects the identifier, name, email, and country from the `customers` table. ```sql SQL theme={null} EXTRACT TO LOCAL OPTIONS( FILE_PREFIX = '/home/user/customer_data/', FILE_TYPE = PARQUET, PARTITION_MODE = 'KEY', PARTITION_COLUMNS = 'country', TARGET_FILE_SIZE_MB = 256 ) AS SELECT id, name, email, country FROM customers; ``` The resulting file structure has these file paths. ``` /home/user/customer_data/country=US/part-0000.parquet /home/user/customer_data/country=CA/part-0000.parquet ... ``` **S3 Parquet Extract with Compression** This example extracts all sales data from the `today_sales` table to a single Parquet file on S3 using ZSTD compression. Use the S3 bucket named `my-analytics-bucket` with the file path `daily_reports/report.parquet` using the endpoint `s3.us-east-1.amazonaws.com`. ```sql SQL theme={null} EXTRACT TO S3 OPTIONS( BUCKET = 'my-analytics-bucket', FILE_PREFIX = 'daily_reports/report.parquet', ENDPOINT = 's3.us-east-1.amazonaws.com', FILE_TYPE = PARQUET, PARQUET_COMPRESSION = 'ZSTD' ) AS SELECT * FROM today_sales; ``` **S3 Parquet Extract with Partitioning by Range** This example extracts large event data to S3 by splitting the `event_id` column into `30` parallel queries using the `RANGE` partition strategy. Specify the maximum file size in `1024` megabytes. Override the default row group size using the `PARQUET_ROW_GROUP_SIZE_BYTES` option to specify `268435456` bytes (256 MB). Use the S3 bucket `iot-data` with the file path `events/2025-09-25/` at endpoint `s3.us-east-1.amazonaws.com`. The SQL query selects the event identifier, sensor identifier, sensor reading, and timestamp from the `events` table. ```sql SQL theme={null} EXTRACT TO S3 OPTIONS( BUCKET = 'iot-data', FILE_PREFIX = 'events/2025-09-25/', ENDPOINT = 's3.us-east-1.amazonaws.com', FILE_TYPE = PARQUET, PARTITION_MODE = 'RANGE', PARTITION_COLUMNS = 'event_id', NUM_FETCH_QUERIES = 30, TARGET_FILE_SIZE_MB = 1024, PARQUET_ROW_GROUP_SIZE_BYTES = 268435456 ) AS SELECT event_id, sensor_id, sensor_reading, ts FROM events; ``` ## Related Links [Connect Using JDBC](/connect-using-jdbc) [JDBC Manual](/jdbc-manual) # Data Formats for Data Pipelines Source: https://docs.ocient.com/data-formats-for-data-pipelines Load JSON, DELIMITED, Avro, Parquet, and other formats into Ocient with data pipelines, including field selectors, type interpretation, and NULL handling. Loading in differs in subtle ways that depend on the data format of the source. Loading runs with a strict interpretation of streaming source with or source data to allow pipelines to achieve maximum performance. For text-based formats like `JSON` and `DELIMITED`, the Ocient System performs no preemptive casting on the data when using a source field selector. For the Kafka streaming source, use the Schema Registry with subjects that contain a group of schema versions, which capture the schema changes over time. For example, with the JSON string `{ "my_field": 1234 }`, the selector `$my_field` returns the string `"1234"` not the integer `1234`. When you use transformation functions, keep in mind that the Ocient System treats all data in the `JSON` and `DELIMITED` formats as text data. While the Ocient System sends data you select to a final target column, the system automatically casts the data in the final step to ensure that the data is compatible with the target column type. See [Data Types for Data Pipelines](/data-types-for-data-pipelines) for supported automatic conversion rules. Format-specific differences also appear in the pipelines. ## Load ASN.1 Data You can load data in ASN.1 (Abstract Syntax Notation One) format from binary-encoded ASN.1 files using DER-encoded or BER-encoded files. ASN.1 provides a flexible, schema-driven format commonly used in telecommunications, security, and standardized protocols. This format allows you to extract structured records and map them to relational tables using SQL. The Ocient System requires `.ber` and `.der` files to contain one or more concatenated DER-encoded or BER-encoded values with the specified record type. The system decodes each record independently and maps it into a record. ### ASN.1 Type Mapping The system automatically converts all decoded ASN.1 fields to their JSON-equivalent representations. The ASN.1 schema must consistently use implicit or explicit tagging. You must specify clear tagging so the system can resolve field names during extraction. | **ASN.1 Type** | **JSON Mapping** | **Description** | | ----------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `INTEGER` | Number | The number, specified as an integer.

Example: `5` | | `BOOLEAN` | Boolean | The logical value, specified as a Boolean.

Example: `true` | | `VisibleString` | String | The ASN.1 type assumes UTF-8 encoding.

Example: `"Data Analyst"` | | `IA5String` | String | The International Alphabet number 5 (IA5) includes most of the ASCII alphabet but can contain other characters.

Example: `"ABCD"` | | `UTF8String` | String | The ASN.1 type assumes UTF-8 encoding.

Example: `"abcd"` | | `OCTET STRING` | Hexadecimal string | The hexadecimal string has the prefix `0x`.

Example: `"0x4fa2"` | | `BIT STRING` | Hexadecimal string | The hexadecimal string with preserved bit alignment has the prefix `0x`.

Example: `"011110"` | | `OBJECT IDENTIFIER` | String | The unique identifier for an object using dotted representation.

Example: `"1.2.840.113549"` | | `ENUMERATED` | Number or symbolic string | The enumerated string, if the data resolves into a list of named items.

Example: `"red"` | | `ANY` | Hexadecimal string | The hexadecimal string with raw encoded bytes of the embedded value.

Example: `"0x616263"` | | `SEQUENCE`, `SET` | JSON object | Object with fields that are accessible by their component names.

Example:
`WeatherRecord ::= SEQUENCE { temperature INTEGER, humidity INTEGER, cloudy BOOLEAN }` | | `SEQUENCE OF`, `SET OF` | JSON array | Array of objects where the elements are fully decoded.

Example:
`WeatherEvents ::= SEQUENCE OF WeatherRecord` | | `CHOICE` | JSON object | An object with fields of any type. Identify the active alternative by the key.

Example:
`WeatherReport ::= CHOICE { sunny SEQUENCE { uv\_index INTEGER }, rainy SEQUENCE { precipitation INTEGER } }` | | `DATE` | String | Date in the format `"yyyyMMdd"`. You must explicitly declare the date, or the system must infer it using tagging.

Example: `"20240518"` | | `TIME` | String | Time in the format `"HHmmss"`.

Example: `"113056"` | | `DATE-TIME` | String | Date and time in the format `"yyMMddHHmmss'Z'"`. This format supports only UTC. You must include `'Z'` as a literal.

Example: `"250920101508'Z'"` | To access fields, use dot notation for access: `$sequence.fieldName`. For arrays (such as `SEQUENCE OF`), use bracket notation: `$sequenceOf[].fieldElement`. Default values and optional fields follow standard ASN.1 rules. If you omit a field, the system evaluates it as `NULL`. ### ASN.1 Loading Example Assume you have the `personnel_records.asn` file in the ASN.1 data format. The file contains the definition of a personnel record `Example.PersonnelRecord`. The ASN.1 file contains data for the personnel record, child, personnel name, employee number of the personnel, and the date. ```none Text theme={null} Example DEFINITIONS IMPLICIT TAGS ::= BEGIN PersonnelRecord ::= [APPLICATION 0] SET { name [0] Name, title [1] VisibleString, number [2] EmployeeNumber, dateOfHire [3] Date, nameOfSpouse [4] Name, children [5] SEQUENCE OF ChildInformation DEFAULT {} } ChildInformation ::= SET { name [0] Name, dateOfBirth [1] Date } Name ::= [APPLICATION 1] SEQUENCE { givenName [0] VisibleString, initial [1] VisibleString, familyName [2] VisibleString } EmployeeNumber ::= [APPLICATION 2] INTEGER Date ::= [APPLICATION 3] VisibleString END ``` Create a table to contain the personnel record. The table contains a subset of the data: * `first_name` — First name * `initial` — Initial of the middle name * `family_name` — Last name * `title` — Job title * `number` — Employee number * `date_of_hire` — Hire date ```sql SQL theme={null} CREATE TABLE personnel_records( first_name VARCHAR NOT NULL, initial VARCHAR NOT NULL, family_name VARCHAR NOT NULL, title VARCHAR NOT NULL, number BIGINT NOT NULL, date_of_hire DATE NOT NULL ); ``` Create the data pipeline `personnel_pipeline` to load the personnel record into the `personnel_records` table using an S3 bucket. Specify the bucket, endpoint, access key identifier, secret access key, and filter options to find the ASN.1 DER-encoded file `personnel_records.der` in the specified file path. Use the URL file path `http://cos/filepath/asn1/personnel_records.asn` and record type `Example.PersonnelRecord`. Access the `name` sequence using dot notation for the first name, middle initial, and last name fields. The pipeline definition transforms the hire date to the `'yyyyMMdd'` format. ```sql SQL theme={null} CREATE PIPELINE personnel_pipeline SOURCE S3 BUCKET 'misc' ENDPOINT 'http://cos' ACCESS_KEY_ID '' SECRET_ACCESS_KEY '' FILTER_GLOB 'user/asn1/personnel_records.der' EXTRACT FORMAT 'asn.1' SCHEMA { URL 'http://cos/filepath/asn1/personnel_records.asn' RECORD_TYPE 'Example.PersonnelRecord' } INTO personnel_records SELECT $name.givenName AS first_name, $name.initial as initial, $name.familyName as family_name, $title as title, $number as number, TO_DATE($dateOfHire, 'yyyyMMdd') as date_of_hire; ``` ## Load Avro Data The Ocient System enables you to load data in the format. You can use a streaming source with a file-based source only. Load an inline schema definition or use a schema configuration. Use a schema inference from files with embedded schemas. Field selectors in Avro follow the same format as selectors in JSON and formats. The Ocient System treats Avro selectors as lowercase. To use case-sensitive selectors, you must enclose the selector in double quotation marks. For example, `$"testSelector"`. ### Inline Schema Specify a JSON string in the Avro schema format in the `INLINE` option of the schema definition of the `CREATE PIPELINE` SQL statement. Inline schemas assume that all records follow the defined schema exactly. These schemas do not support schema evolution. For Kafka messages, use the `SCHEMA_REGISTRY_ID_LOCATION` option to denote whether there is an embedded schema identifier. For inline schemas, the `'none'` value indicates that no embedded schema identifier is present. ### Schema Inference from a File The system can infer the schema from a file that has embedded schemas. The file is a named object container file. Use the `INFER_FROM` option in the `CREATE PIPELINE` SQL statement to specify sampling one file. ### Schema Registry For loading from a Kafka streaming source, use the schema configuration with the `SUBJECT` option in the `CREATE PIPELINE` SQL statement to specify the name of the subject for the data pipeline. The default value is `-value`, where `` is the name of the Kafka topic. The Ocient System follows the schema registry configuration from the Confluent platform. ### Schema Evolution When you create a data pipeline, the pipeline has a fixed target schema (specified by the `SELECT` clause). Individual files might have different schemas. The target schema must be backward-transitive compatible with the other schemas. Individual files or Kafka messages might have different schemas. If the other schemas change, the system automatically attempts to fit data into the target schema. In this case, the other schemas must be forward compatible with the target schema. The system ignores changes to any unused fields from the target schema. To manually change the schema, you must first stop the execution of the data pipeline by using the `STOP PIPELINE` SQL statement. Then, use the `ALTER PIPELINE` SQL statement to modify the schema. You can add a field, remove a field, or change the precision of a data type. For details, see the [ALTER PIPELINE](/data-pipelines#alter-pipeline) SQL statement. Multiple schemas impact the performance of the data pipeline execution. For best performance, use a single schema for all data. ### Avro Type Mapping The Ocient System converts these Avro data types to Ocient SQL types. This table shows the respective conversions. | **Avro Type** | **Ocient SQL Type** | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `null` | `NULL` | | `boolean` | `BOOLEAN` | | `int` | `INT` | | `long` | `BIGINT` | | `float` | `FLOAT` | | `double` | `DOUBLE` | | `bytes` | `VARBINARY` | | `string` | `VARCHAR` | | `record` | Select subfields of the record using the standard pipeline selector syntax, e.g., `$myrecord.subfield`. | | `enum` | `VARCHAR` | | `array` | `ARRAY` | | `map` | Select entries of the map using the key with the standard pipeline selector syntax, e.g., `$mymap.key`. | | `fixed` | `HASH` | | `decimal` | `DECIMAL` | | `big-decimal` | `DECIMAL` | | `uuid` | `UUID` | | `date` | `DATE` | | `time-*` | `TIME` | | `timestamp-*` | `TIMESTAMP` | | `local-timestamp-*` | `TIMESTAMP` | | `duration` | The system converts duration to a record with integer subfields for `months`, `days`, and `milliseconds`. | | `union` | The system converts a union between `NULL` and another type to a nullable version of that type. In contrast, the system converts a union containing multiple non-`NULL` types to a string. | ### Avro Loading Examples Create the `users` table with these columns: * `id` — Universally Unique IDentifier (UUID) of the user * `firstname` — First name of the user * `lastname` — Last name of the user * `birthyear` — Year of birth * `groups` — List of groups where the user belongs ```sql SQL theme={null} CREATE TABLE users( id UUID NOT NULL, firstname VARCHAR(255) NOT NULL, lastname VARCHAR(255) NOT NULL, birthyear INT, groups VARCHAR(255)[] NOT NULL DEFAULT 'char[]' ); ``` These examples use this table as the target table for the load. **Load Avro Data from Files** Assume you have user data in Avro format in multiple files in the `/data/users` directory. Create the `users_pipeline` data pipeline for the Avro files `*.avro` containing user data. The schema configuration instructs the system to infer from one file using the `INFER_FROM` option. Access the array of strings for the `groups` column. ```sql SQL theme={null} CREATE PIPELINE users_pipeline SOURCE filesystem FILTER '/data/users/*.avro' EXTRACT FORMAT avro SCHEMA { INFER_FROM 'sample_file' } INTO users SELECT $id AS id, $firstname AS firstname, $lastname AS lastname, $birthyear AS birthyear, $groups[] AS groups; ``` **Load Avro Data from an Inline Schema Definition** Assume you have user data in Avro format in multiple files in the `/data/users` directory. Create the `users_pipeline` data pipeline for the Avro files `*.avro` containing user data. The schema configuration instructs the system to use an inline schema definition with the `INLINE` option. Access the array of strings for the `groups` column. ```sql SQL theme={null} CREATE PIPELINE users_pipeline SOURCE filesystem FILTER '/data/users/*.avro' EXTRACT FORMAT avro SCHEMA { INLINE '{ "type": "record", "name": "User", "namespace": "test.users", "fields": [ { "name": "id", "type": { "type": "string", "logicalType": "uuid" } }, { "name": "firstname", "type": "string" }, { "name": "lastname", "type": "string" }, { "name": "birthyear", "type": ["null","int"], "default": null }, { "name": "groups", "type": { "type": "array", "items": "string" } } ] }' } INTO users SELECT $id AS id, $firstname AS firstname, $lastname AS lastname, $birthyear AS birthyear, $groups[] AS groups; ``` **Load Avro Data Using a Kafka Schema Registry Configuration** Assume you have user data in Avro format in a Kafka topic. Create the `users_pipeline` data pipeline for the user data. The schema registry configuration instructs the system to access the subject `'users-value'` at the registry location `'value'` using the registry URL `'https://schema-registry.company.com'`. The configuration specifies the access credentials using the `CONFIG` option: * Credentials source * User authentication that includes the username and password * Location of the truststore * Password for the truststore Access the array of strings for the `groups` column. ```sql SQL theme={null} CREATE PIPELINE users_pipeline SOURCE KAFKA BOOTSTRAP_SERVERS '192.168.0.1:9092' TOPIC 'users' EXTRACT FORMAT avro SCHEMA { SUBJECT 'users-value' SCHEMA_REGISTRY_ID_LOCATION 'value' URL 'https://schema-registry.company.com' CONFIG '{ "basic.auth.credentials.source": "USER_INFO", "basic.auth.user.info": "sr_user:sr_password", "ssl.truststore.location": "/etc/ssl/truststore.jks", "ssl.truststore.password": "testpassword" }' } INTO users SELECT $id AS id, $firstname AS firstname, $lastname AS lastname, $birthyear AS birthyear, $groups[] AS groups; ``` For details about the schema registry configuration, see the [Confluent Documentation](https://docs.confluent.io/). ## Load Binary Data The Ocient System loads the binary data format using a fixed record length to split a binary stream into chunks that represent records. Each record is available in the `SELECT` portion of a pipeline definition using a special binary extract syntax `$"[5,8]"`. This operates similarly to a substring function, beginning at byte `5` and taking `8` bytes from that location. The starting index is a 1-based offset, consistent with other SQL arrays and offsets. You can use this syntax to select specific bytes within a record to parse together as a unit. ### Binary Selector | **Selector** | **Description** | **Examples** | | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `$"[,]"` | **Extract Bytes** — Extracts the bytes beginning at the `` and returning `` total bytes to extract from the start index.
`` is 1-based. | `abc123.45xyz`
`$"[4,6]" -> "123.45"`
This example illustrates the operation using ascii characters, but the Binary Selector returns BINARY data, byte arrays, that may be used in special ways by fixed width processing functions described in [Binary Data Transformations](/transform-data-in-data-pipelines#binary-data-transformation-functions). | **Example** The Binary Selector takes 8 bytes starting at offset 11 on the fixed-width binary record. Consistent with SQL functions in the Ocient System, the first argument value `11` is the 1-based offset into the byte array. ```sql SQL theme={null} $"[11, 8]" ``` The Binary Selector returns `BINARY` data, not `VARCHAR`. Special `BINARY` transformation functions can operate on this `BINARY` data. However, if you cast data to the `VARCHAR` type by using `CHAR()`, then functions like `INT` operate on this data as `VARCHAR` data, not binary data. When you load binary data into `VARCHAR` columns, the Ocient System automatically converts from binary to character data using the configured `CHARSET_NAME` before final loading. The Ocient System supports special transformation functions that operate uniquely on binary data. With these functions, you can convert binary representations from mainframe systems such as packed decimals, zoned decimals, big and little endian integers (signed and unsigned), and floating point values. For more details, see [Binary Data Transformation Functions](/transform-data-in-data-pipelines). For a complete list of supported options for `DELIMITED` and `CSV` data formats, see [Binary Extract Options](/data-pipelines#binary-extract-options). ### Binary Loading Example If each record in your fixed-width binary schema includes these fields, you can use the `SUBSTRING` function and the transforms shown in this example. | **Column** | **Start Index** | **Record Length** | **Source Data Type** | **Ocient Target Data Type** | | ------------ | --------------- | ----------------- | ---------------------------------- | --------------------------- | | first\_name | 1 | 20 | Character | VARCHAR | | last\_name | 21 | 20 | Character | VARCHAR | | age | 41 | 4 | Big Endian signed 4-byte integer | INT | | total\_spent | 45 | 10 | Packed Decimal | DECIMAL(10,2) | | user\_id | 55 | 8 | Little Endian unsigned 8-byte long | BIGINT | Each record includes 62 bytes, so the record length `RECORD_LENGTH` is 62. The encoding of this file is CP500 instead of the default IBM1047 code page. The `CREATE PIPELINE` SQL statement specifies this encoding. ```sql SQL theme={null} CREATE PIPELINE binary_users_pipeline SOURCE S3 ... FORMAT BINARY RECORD_LENGTH 62 CHARSET_NAME 'cp500' INTO public.users SELECT $"[1, 20]" as first_name, $"[21, 20]" as last_name, INT($"[41, 4]") as age, DECIMAL($"[45, 10]", 'packed', 2) as total_spent, BIGINT($"[55, 8]", 'unsigned', 'little') as user_id; ``` This SQL statement: * Uses the `BINARY SELECTOR` to extract names and load them into the respective columns. The Ocient System automatically decodes the values using `cp500` and loads them into a `VARCHAR` column. An explicit cast such as `CHAR($"[1, 20]") as first_name` works equivalently. * Indicates the extraction of four bytes that represent age from bytes 41-44. The statement instructs the casting of these bytes as an integer `INT`. This function uses the default endianness (big) and treats the bytes as signed. Unsigned values can overflow target columns because integral types are all signed. * Extracts the 10 bytes for `total_spent` using the `BINARY SELECTOR`, and converts the values using the `packed` decimal option for the `DECIMAL` cast. The casting requires specifying the number of decimal points in the source data. In this case, there are 2 decimal points, which match the number in the target column. * Extracts the 8 bytes that represent `user_id` using the `BINARY SELECTOR` and casts these bytes to a `BIGINT` while interpreting the bytes as unsigned with the little endian representation. ## Load Delimited and CSV Data When you load data from delimited or CSV files, the Ocient System tokenizes the data during loading. The system detects records and fields in the input data during pipeline execution. You can reference fields and use them in combination with transformation functions before the system stores values in the column of a target table. Files must be located in these allowed directories: * `/tmp` directory * The temporary directory you configure using the `streamloader.extractorEngineParameters.tempDir` configuration option (default path is `/var/opt/ocient/tmp`). * The directory list specified by the `streamloader.extractorEngineParameters.configurationOption.filesystem.access.directories `configuration option (default is an empty list). Files cannot be located in these blocked directories: * `/etc` * `/bin` * `/sbin` * `/lib` * `/lib64` * `/usr` * `/boot` * `/proc` * `/sys` * `/run` * `/root` * `/var/lib` * `/var/log` * `/var/run` * `/var/cache` Referencing fields of the source data for the formats happens by using a field index. The index is a number that follows the dollar sign `$`. To maintain consistency with SQL array semantics, the field indexes start at 1. Reference the first field of tokenized source records for the `DELIMITED` and `CSV` formats as `$1`. For the `BINARY` format, `$0` represents the entire record. In this case, you must specify `$0` in combination with the `SUBSTRING` function to extract specific bytes from the source data. For a complete list of supported options for `DELIMITED` and `CSV` data formats, see [Delimited and CSV Extract Options](/data-pipelines#delimited-and-csv-extract-options). ### Delimited Loading Example Use this example delimited data. ```none Text theme={null} iphone|60607|viewed|502|293.99|[shopping,news] ``` For this example row, this table shows the field references for each value in the row. | **Field Reference** | **Value** | | ------------------- | ---------------------- | | \$1 | iphone | | \$2 | 60607 | | \$3 | viewed | | \$4 | 502 | | \$5 | 293.99 | | \$6\\\[] | \\\['shopping','news'] | To load this data in a pipeline with the `DELIMITED` data format, this `CREATE PIPELINE` statement specifies the `|` character for the field delimiter. This statement loads data into AWS S3. The `SELECT` statement uses fields 1, 2, 3, 5, and 6 of the source data. The statement specifies that the system should not load field 4 to the target table. Field 6 is an array of data matching the default array settings for delimited data. You can indicate this with the array brackets like `$6[]` to load into a `CHAR[]` typed column. The outer casting functions in this example are optional and shown for completeness. If they are omitted, the pipeline automatically casts the source fields to the target column type. ```sql SQL theme={null} CREATE PIPELINE delimited_pipeline SOURCE s3 ... FORMAT delimited FIELD_DELIMITERS ['|'] ... SELECT CHAR($1) as device_model, INT($2) as zip, INT($3) as amount, DOUBLE($5) as price, CHAR[]($6[]) as categories; ``` ## Load JSON Data The data pipeline syntax enables the load of JSON data, including nested scalars, arrays, and points `ST_POINT`. **Strict Loading and Transformations** When you use transformation functions, remember that the Ocient System treats all data in `JSON` and `DELIMITED` format as text data, not the `logical` data type. For example, if you specify the JSON string `{ "my_timestamp": 1709208000000 }`, the selector `$my_timestamp` returns the string `"1709208000000"` and not the integer `1709208000000`. As a result, if you cast this data into a timestamp column, such as `TIMESTAMP($my_timestamp) as created_at`, the Ocient System returns an error. The conversion fails because the cast function assumes you are specifying `TIMESTAMP(VARCHAR)`, which assumes a format like `YYYY-MM-DD HH🇲🇲ss[.SSSSSSSSS]`. To correct this issue, cast the value explicitly to make use of the `TIMESTAMP(BIGINT)` function that treats the argument as milliseconds after the epoch as in `TIMESTAMP(BIGINT($my_timestamp)) as created_at`. ### Supported JSON Selectors JSON selectors consist of `$` followed by a dot-separated list of JSON keys. If a key refers to an array, it is followed by a set of brackets `[]` to correspond to its dimensionality. If the square brackets contain an index, like `[1]`, then the selector refers to an array element. The Ocient System treats JSON selectors as lowercase. To use case-sensitive selectors, you must enclose the selector in double quotation marks. For example, `$"testSelector"`. With case-sensitive selectors having multiple JSON keys, each key needs double quotation marks. For example, `$"testData"."Responses"."SuccessResponse"`. For special characters (any identifier that starts with any character other than a letter or contains any character that is not a letter, number, or an underscore) or reserved SQL keywords (such as `SELECT`), you must enclose such selectors in double quotation marks. For example, if you have a JSON document `{ "test-field": 123 }`, then the selector for the query should be `$"test-field"`. The Ocient System does not support identifiers with a backslash as the last character in the key name. This table shows the selector and provides its description. The cells in the last column of the table show an example for each selector. First, the cell shows example data in JSON format. Then, the cell shows the use of the selector and its output after the arrow. | **Selector** | **Description** | **Examples** | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `$a` | **Scalar** — Refers to key `a` at the top of a JSON document, where the value of `a` is a scalar. | `{ "first_name": "Chris" }`

`$first_name -> "Chris"` | | `$a[]` | **Array** — Refers to key `a` at the top of a JSON document, where the value of `a` is a one-dimensional array. | `{ "names": ["Chris", "Joe"] }`
`$names[] -> ["Chris","Joe"]` | | `$a[1]` | **Array Element Selection** — Selects a single element from the array `a` at the top of a JSON document, where the value of `a` is a one-dimensional array. Also works on multi-dimensional arrays. Indexes start at 1. | `{ "names": ["Chris", "Joe"] }`
`$names[2] -> "Joe"` | | `$a[][]` | **Multi-dimensional Array** — Refers to key `a` at the top of a JSON document, where the value of `a` is a two-dimensional array. | `{ "coordinates": [ [87.31, 18.25],[65.22, 19.41]] }`
`$coordinates[][] -> [[87.31,18.25],[65.22,19.41]]` | | `$a.b.c` | **Nested Selector** — Drills into the value at key `a`, then value at key `b`, and refers to the value at key `c`, where none of the values are arrays. | `{ "user": { "name": "Chris" } }`
`$user.name -> "Chris"` | | `$a[].b` | **Array Projection** — A projection applies a JSON selector on all elements in an array, returning an array.
Array brackets in the notation indicate which items are the arrays.
The `a[]` is an array. The `b` selector is applied to each element in the array `a`. | `{`
`"users": [`
`{`
`"name": "Chris",`
`"orders": [`
`{ "order_id": 1,`
`"subtotal": 19.29 },`
`{ "order_id": 2,`
`"subtotal": 16.71 }`
`]`
`},`
`{`
`"name": "Joe",`
`"orders": [`
`{ "order_id": 3,`
`"subtotal": 17.29 },`
`{ "order_id": 4,`
`"subtotal": 22.49 }`
`]`
`}`
`]`
`}`
`$users[].name -> ["Chris", "Joe"]` | | `$a.b[].c.d[]` | **Multi-level Array Projection** — Applies selectors to each element in the specified arrays and allows multiple levels of objects and arrays for the selection of an N-dimensional array of values.
The values of `b` and `d` are arrays, resulting in two-dimensional arrays. The selector `c.d[]` is projected into the elements in the array `b`. | `{`
`"users": [`
`{`
`"name": "Chris",`
`"orders": [`
`{ "order_id": 1,`
`"subtotal": 19.29 },`
`{ "order_id": 2,`
`"subtotal": 16.71 }`
`]`
`},`
`{`
`"name": "Joe",`
`"orders": [`
`{ "order_id": 3,`
`"subtotal": 17.29 },`
`{ "order_id": 4,`
`"subtotal": 22.49 }`
`]`
`}`
`]`
`}`
`$users[].orders[].order_id -> [[1,2],[3,4]]` | | `$a.b[1].c.d[]` | **Array Slice** — Selects the first element from the array `b` and drills into `c` and then the array `d`. JSON array selectors have an index that starts at 0. | `{`
`"users": [`
`{`
`"name": "Chris",`
`"orders": [`
`{ "order_id": 1,`
`"subtotal": 19.29 },`
`{ "order_id": 2,`
`"subtotal": 16.71 }`
`]`
`},`
`{`
`"name": "Joe",`
`"orders": [`
`{ "order_id": 3,`
`"subtotal": 17.29 },`
`{ "order_id": 4,`
`"subtotal": 22.49 }`
`]`
`}`
`]`
`}`
`$users[1].orders[].subtotal -> [17.29,22.49]` | | `$a.{b,c}` | **Tuple Selector** — Selects the values from an object as a tuple. You must cast the values to an appropriate `TUPLE` data type. | `{ "user": { "first_name": "John", "last_name": "Doe" }}`
`$user.{first_name,last_name} -> <<"John","Doe">>` | | `$a[].{b,c}` | **Array of Tuples Selector** — Selects the values from an array of objects as an array of tuples. You must cast the values to an appropriate `TUPLE[]` data type. | `{ "users": [ { "first_name": "John", "last_name": "Doe" }, { "first_name": "Steve", "last_name": "Smith" } ]}`
`$users[].{first_name,last_name} -> [<<"John","Doe">>,<<"Steve","Smith">>]` | | `$a[_]` | **Flatten Array Selector** — Selects the values of an array and flattens the dimension indicated by the underscore by one level. | `{`
`a: {`
`b: [`
`{c: [1,2,3] },`
`{c: [4,5,6] }`
`]`
`}`
`}`
`$a.b[_].c -> [1,2,3,4,5,6]` | | `$a[!]` | **Compact Array Selector** — Selects the values of an array and removes any NULL values from the array at this level in the array. | `{`
`a: {`
`b: {`
`c: [1,null,2,null,3]`
`}`
`}`
`}`
`$a.b.c[!] -> [1,2,3]` | For more examples of using JSON selectors in data pipelines, see [JSON Selectors Examples in Data Pipelines](/json-selectors-examples-in-data-pipelines). ### NULL and Empty Handling for JSON Scalars The Ocient System handles all JSON NULL, empty, and missing values in the same way. The system loads these values as `NULL`. These values fail to load into non-nullable columns. Provide an explicit default in the pipeline using `IF_NULL` or `COALESCE` or use the `COLUMN_DEFAULT_IF_NULL` option to accept the configured column default instead of attempting to load `NULL` values. ### NULL and Empty Handling for JSON Arrays The Ocient System handles NULL, empty, and missing values the same way for arrays as for scalars. The system converts a value that is NULL, empty, or missing to NULL and loads it as `NULL`. Provide an explicit default in the pipeline or use the `COLUMN_DEFAULT_IF_NULL` option to accept the configured column default instead of attempting to load `NULL`. ### NULL and Empty Handling for JSON Tuples All the rules for handling NULL, empty, and missing elements that apply to scalars and arrays also apply to tuples. If any part of the selector is NULL, empty, or missing, data pipeline loading converts that value to `NULL`. Additionally, because you can apply functions to tuple elements (and not array elements), you can use the `NULL_IF` function to convert a tuple element to `NULL`. For example, `tuple<>($a.name, NULL_IF($a.hometown, 'N/A') )` indicates to the pipeline that the string `'N/A'` signifies `NULL` for the `hometown` element but not for the `name` element. ## Load Parquet Data The data pipeline functionality enables loading Parquet files with this configuration. **File Configuration** * Files should have row groups of less than 128 MB. Larger row groups can impact memory usage during loading, and row groups of 512 MB can cause loading failures on 1 TB or more data sets. * Encoding fields in a Parquet file reduces the space of the file on disk but can impact memory usage during loading. Enable encoding on fields that you expect to have less than 256 unique values and for fields that contain short strings. You do not have to encode other fields. **Multiple Files** * You can load row groups of multiple Parquet files in parallel. For large data sets, load the data set as multiple files. * Loading files with differing schemas is not supported. Use selectors as you do when loading JSON data to specify data to load. You must select a leaf element, an array, or a tuple with your selector. This is stricter than using JSON selectors, which can directly select array fields and JSON object fields. Example: `{"a": [1,2,3], "b": {"c": 1}}` You can extract with any of the selectors in JSON: `$a, $a[], $b, $b.c` However, Parquet only allows for the selectors: `$a[], $b.c` This example assumes this schema: ``` // List (list non-null, elements nullable) required group my_list (LIST) { repeated group list { optional binary element (UTF8); } } ``` The selector must be `$my_list[]`, which includes the array syntax. For details, see [Parquet Selectors Examples in Data Pipelines](/parquet-selectors-examples-in-data-pipelines). When you use the `FORMAT PARQUET` option with an AWS S3 source, the `ENDPOINT` option is required in the `CREATE PIPELINE` SQL statement. Auto-casting in Parquet does not support the automatic conversion to `VARCHAR` columns. You must explicitly cast data to the `CHAR` data type when you convert Parquet data that is not string data to a `VARCHAR` column or `VARCHAR` function argument. The Ocient System treats Parquet selectors as lowercase. To use case-sensitive selectors, you must enclose the selector in double quotation marks. For example, `$"testSelector"`. ### Schema Evolution The Ocient System supports schema evolution when you load a set of Parquet files. Specifically, if the pipeline selects a set of Parquet files where an individual file might have more or fewer columns than another, the system attempts to merge those schemas together to support loading without requiring you to create the pipeline again. For example, the `test_table` table has three columns. ```sql SQL theme={null} CREATE TABLE test_table ( col_a INT NULL, col_b VARCHAR NULL, col_c VARCHAR NULL ); ``` You have two Parquet files with these schemas: ```none Text theme={null} message file1_schema { OPTIONAL INT32 col_a; OPTIONAL BYTE_ARRAY col_b (UTF8); } message file2_schema { OPTIONAL BYTE_ARRAY col_b (UTF8); OPTIONAL BYTE_ARRAY col_c (UTF8); } ``` However, you must specify how to handle the schema evolution within the `EXTRACT` SQL statement. You can choose to sample the first file only for its schema or sample the entire data set to merge the schemas together. This DDL statement samples on one file. ```sql SQL theme={null} EXTRACT FORMAT parquet SCHEMA (INFER_FROM sample_file) ``` The disadvantage is that sampling multiple files can potentially take a long time (scaling with the number of files in the data set) when you execute the `CREATE PIPELINE` and `START PIPELINE` SQL statements. If you know that all of the Parquet files have the same schema, use this syntax. The Ocient System does not support the case where a column within the schema changes type. For example, if `col_a` is an `INT` type in one file and a `VARCHAR` type in another. The default behavior of schema evolution infers the schema from one file. Use this syntax to infer from one file. ```sql SQL theme={null} EXTRACT FORMAT parquet SCHEMA (INFER_FROM sample_file) ``` ### Parquet Type Mapping Parquet data types are separated into primitive and logical types. The Ocient System converts these types to Ocient SQL types. See these tables for the respective conversions. | **Parquet Primitive Type** | **Ocient SQL Type** | | -------------------------- | ------------------- | | `BOOLEAN` | `BOOLEAN` | | `INT32` | `INT` | | `INT64` | `BIGINT` | | `INT96` | `TIMESTAMP` | | `FLOAT` | `FLOAT` | | `DOUBLE` | `DOUBLE` | | `BYTE_ARRAY` | `VARCHAR` | | `FIXED_LEN_BYTE_ARRAY` | `VARCHAR` | | **Parquet Logical Types** | **Ocient SQL Type** | | ------------------------- | ------------------- | | `STRING` | `VARCHAR` | | `UTF8` | `VARCHAR` | | `ENUM` | `VARCHAR` | | `UUID` | `UUID` | | `INT8` | `TINYINT` | | `INT16` | `SMALLINT` | | `INT32` | `INT` | | `INT64` | `BIGINT` | | `UINT8` | `SMALLINT` | | `UINT16` | `INT` | | `UINT32` | `BIGINT` | | `UINT64` | `BIGINT` | | `DECIMAL` | `DECIMAL` | | `DATE` | `DATE` | | `TIME` | `TIME` | | `TIME_MILLIS` | `TIME` | | `TIME_MICROS` | `TIME` | | `TIMESTAMP` | `TIMESTAMP` | | `TIMESTAMP_MILLIS` | `TIMESTAMP` | | `TIMESTAMP_MICROS` | `TIMESTAMP` | | `DURATION` | `BIGINT` | | `JSON` | `VARCHAR` | | `BSON` | `VARCHAR` | * The `INTERVAL` data type is not supported. * The `UINT64` data type can overflow the `BIGINT` conversion. * The `DURATION` data type conversion to `BIGINT` preserves the underlying units. For example, the number of microseconds stays as microseconds in the `BIGINT` data type. Further, Parquet contains nested types that the Ocient System also converts to SQL types, as shown in this table. | **Parquet Nested Types** | **Ocient SQL Type** | | ------------------------ | ------------------- | | `LIST` | `TYPE[]` | | `TUPLE` | `TUPLE` | ### Parquet Loading Example Create a data pipeline that loads Parquet files using an AWS S3 bucket. Specify the bucket, endpoint, access key identifier, secret access key, and filter options to find all Parquet files in the specified file path. Use the `parquet_base_table` table to store the loaded data. Retrieve the integer, text, floating point, double, integer, JSON, and BSON fields. ```sql SQL theme={null} CREATE PIPELINE testpipeline SOURCE S3 BUCKET 'testbucket' ENDPOINT 'https://endpoint' ACCESS_KEY_ID '' SECRET_ACCESS_KEY '' FILTER_GLOB '/data/*/2024/*/*.parquet' PREFIX '/data/orders/2024/11/' EXTRACT FORMAT PARQUET INTO parquet_base_table SELECT $int32_field AS int32_field, $utf8_field as utf8_field, $float_field as float_field, $double_field as double_field, $int64_field as int64_field, $json_field as json_field, $bson_field as bson_field; ``` ### Parquet File-Partitioned Data With Parquet, you can load file-partitioned data from Parquet files using the file path structure. Use the filter set in the file path using the naming standards. Assume files with these file paths. ```shell Shell theme={null} s3://data/orders/2024/11/dt=2024-11-24/file.parquet .../dt=2024-11-25/file.parquet .../dt=2024-11-26/file.parquet ... ``` Load the data values in the Parquet file partitions using the `METADATA` function with the Hive partition syntax and the specified partition key `dt` from the file paths. ```sql SQL theme={null} SELECT int32_field, utf8_field, float_field, double_field, int64_field, json_field, bson_field, METADATA('hive_partition','dt') AS file_date; ``` For details about this syntax, see [Load Metadata and File-Based Partitioned Data in Data Pipelines](/load-metadata-and-file-based-partitioned-data-in-data-pipelines). ## Load XML Data You can load data in XML format into the Ocient System. The system supports XML tags, basic elements, nested elements, and CDATA, but it does not support XML arrays and attributes. ### Supported XML Selectors Like JSON selectors, you can select XML data using the `$` symbol followed by a list of dot-separated JSON keys. The system treats JSON selectors as lowercase. To use case-sensitive selectors, you must enclose the selector in double quotation marks, such as `$"testSelector"`. With case-sensitive selectors having multiple JSON keys, each key needs double quotation marks, such as `$"testData"."Responses"."SuccessResponse"`. The system does not support JSON array and tuple selectors. ### XML Loading Example Assume an XML file with this data. ```xml XML theme={null} Barbara Smith
New York 12345
127.0.0.1]]> ``` Create a table to contain the IP address record: * `name` — Name * `city` — Name of the city * `zip` — Zip code * `personal_ip` — IP address ```sql SQL theme={null} CREATE TABLE example_xml( name VARCHAR NOT NULL, city VARCHAR NOT NULL, zip VARCHAR NOT NULL, personal_ip IPV4 NOT NULL ); ``` Create a data pipeline `xml_pipeline` that loads the XML file `test.xml` using an AWS S3 bucket. Specify the bucket, endpoint, access key identifier, secret access key, and filter options to find the XML file in the specified path. Use the `example_xml` table to store the loaded data. Use JSON selectors to parse the file and data in each XML tag. The system parses the `CDATA` section in the `note` tag as the literal string `Personal IP 127.0.0.1`. Use the `SUBSTRING` function to extract the IP address and then transform it into an IPV4 type with the `IPV4` function. For details, see the [SUBSTRING](/character-and-binary-functions#substring) and [IPV4](/network-type-functions#ipv4) functions. ```sql SQL theme={null} CREATE PIPELINE xml_pipeline SOURCE S3 BUCKET 'testbucket' ENDPOINT 'https://endpoint' ACCESS_KEY_ID '' SECRET_ACCESS_KEY '' FILTER_GLOB '/data/text.xml' EXTRACT FORMAT XML INTO example_xml SELECT $"root"."person"."name" as name, $"root"."address"."city" as city, $"root"."address"."zip" as zip, IPV4(SUBSTRING($"root"."person"."note", 16, 9)) as personal_ip; ``` ## Related Links [JSON Selectors Examples in Data Pipelines](/json-selectors-examples-in-data-pipelines) [Data Pipelines Reference](/data-pipelines) [Load Metadata and File-Based Partitioned Data in Data Pipelines](/load-metadata-and-file-based-partitioned-data-in-data-pipelines) # Data Integrity and Storage Source: https://docs.ocient.com/data-integrity-and-storage How Ocient maintains data integrity at exabyte scale with checksums, segment redundancy, parity, fault tolerance, and storage cluster configuration. provides several capabilities to check on the quality of data on a node and to quarantine segment groups that are problematic for any reason. Also, you can manage pages by converting them to segments. Related to data integrity is the ability to rebuild tasks. For details, see [Distributed Tasks](/distributed-tasks). ## ALTER SEGMENT QUARANTINE `ALTER SEGMENT QUARANTINE` isolates segment groups from segment generation or queries. Use this SQL statement if segments are causing unexpected results or crashes. The segments can be un-quarantined by setting `quarantine_level` to `none`.  **Syntax** ```sql SQL theme={null} -- To quarantine a single segment group: ALTER SEGMENT QUARANTINE FOR { } WHERE SEGMENT_GROUP_ID = seg_id -- To quarantine multiple segment groups: ALTER SEGMENT QUARANTINE FOR { } WHERE SEGMENT_GROUP_ID in ( seg_id [,...] ) ::= segment_generation | query | all | none ``` | **Parameter** | **Data type** | **Description** | | ------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `seg_id` | numeric | The identification numbers for segment groups.
Depending on your `ALTER SEGMENT QUARANTINE` statement, the Ocient System quarantines the specified segment groups for having an existing quarantine removed.
For details about finding abnormal segments, see the [Guide to Rebuilding Segments](/guide-to-rebuilding-segments). | ### Using Quarantine Levels ( `` ) You can place quarantines at these segment levels. | **Level** | **Description** | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `segment_generation` | Isolates the specified segments from segment generation.
This statement prevents corrupt data from causing problems while still leaving the data available for future analysis. | | `query` | Isolates segment groups from subsequent queries.
Data that is quarantined at the query level is not returned by database queries. This can be useful to prevent corrupt data from causing query problems while leaving it available for future analysis. | | `all` | Isolates the specified segments from segment generation and subsequent queries. | | `none` | Removes any quarantines from the specified segments. | **Examples**  This example quarantines a single segment group from `segment_generation`.  ```sql SQL theme={null} ALTER SEGMENT QUARANTINE FOR segment_generation WHERE SEGMENT_GROUP_ID = 123456789; ``` This example quarantines multiple segment groups from `segment_generation`.  ```sql SQL theme={null} ALTER SEGMENT QUARANTINE FOR segment_generation WHERE SEGMENT_GROUP_ID IN (1,2,3,4,5); ``` This example removes a quarantine from a single-segment group. ```sql SQL theme={null} ALTER SEGMENT QUARANTINE FOR none WHERE SEGMENT_GROUP_ID = 123456789; ``` This example removes a quarantine from multiple segment groups. ```sql SQL theme={null} ALTER SEGMENT QUARANTINE FOR none WHERE SEGMENT_GROUP_ID IN (1,2,3,4,5); ``` This example quarantines at the `query` level. ```sql SQL theme={null} ALTER SEGMENT QUARANTINE FOR query WHERE SEGMENT_GROUP_ID = 123456789; ``` This example removes a quarantine at the `query` level. ```sql SQL theme={null} ALTER SEGMENT QUARANTINE FOR none WHERE SEGMENT_GROUP_ID = 123456789; ``` This example quarantines at both the query and segment-group levels. ```sql SQL theme={null} ALTER SEGMENT QUARANTINE FOR all WHERE SEGMENT_GROUP_ID = 123456789; ``` ## DRAIN PAGES The `DRAIN PAGES` SQL statement instructs the Ocient System to immediately convert buffered pages into segments on one or more Loader Nodes. During normal operation, the system converts pages to segments automatically based on internal thresholds and timeouts, so the `DRAIN PAGES` statement is usually needed only in special circumstances. For example, draining pages can be useful before performing planned maintenance on Loader Nodes. The SQL statement targets all Loader Nodes by default. You can optionally scope the operation to a specific storage scope or table, and you can restrict it to a subset of Loader Nodes. The `DRAIN PAGES` statement forces the immediate restructuring of buffered data into durable segments, which can affect system resource utilization and loading performance. Contact Ocient Support before using this SQL statement. **Required Privileges** * To drain all pages or drain pages for a specific storage scope, you must have `UPDATE` privileges on the system. * To drain pages for a specific table, you must have `UPDATE` privileges on the system or `DELETE` privileges on the database that contains the table. **Syntax** ```sql SQL theme={null} DRAIN PAGES [ FOR { SCOPE UUID = scope_uuid | TABLE UUID = table_uuid } ] [ WITH LOADERS loader_name [, loader_name [, ...] ] ] ``` | **Parameter** | **Type** | **Description** | | ------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `scope_uuid` | string | The Universally Unique IDentifier (UUID) of a storage scope. When you specify this value, the system converts only pages that belong to this storage scope. This value is mutually exclusive with the `table_uuid` value. To find storage scope UUIDs, query the [sys.storage\_scopes](/system-catalog#sys-storage_scopes) system catalog table. | | `table_uuid` | string | The UUID of a table. When you specify this value, the system converts only pages that belong to this table. This value is mutually exclusive with the `scope_uuid` value. To find table UUIDs, query the [sys.tables](/system-catalog#sys-tables) system catalog table. | | `loader_name` | string | The name of a Loader Node. When you specify one or more loader names, the system sends the drain request only to those nodes.

By default, the system sends the request to all Loader Nodes. | **Examples** **Drain All Pages** This example converts all buffered pages to segments across every Loader Node. ```sql SQL theme={null} DRAIN PAGES; ``` **Drain Pages for a Storage Scope** This example converts only pages associated with the storage scope identified by the UUID `a1b2c3d4-e5f6-7890-abcd-ef1234567890`. ```sql SQL theme={null} DRAIN PAGES FOR SCOPE UUID = 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'; ``` **Drain Pages for a Table** This example converts only pages associated with the table identified by the UUID `12345678-abcd-ef01-2345-6789abcdef01`. ```sql SQL theme={null} DRAIN PAGES FOR TABLE UUID = '12345678-abcd-ef01-2345-6789abcdef01'; ``` **Drain Pages on Specific Loader Nodes** This example sends the drain request to the `loader-node-1` and `loader-node-2` Loader Nodes. ```sql SQL theme={null} DRAIN PAGES WITH LOADERS 'loader-node-1', 'loader-node-2'; ``` **Drain Pages for a Table on a Specific Loader Node** This example converts only pages associated with the table identified by the UUID `12345678-abcd-ef01-2345-6789abcdef01` on the `loader-node-1` Loader Node. ```sql SQL theme={null} DRAIN PAGES FOR TABLE UUID = '12345678-abcd-ef01-2345-6789abcdef01' WITH LOADERS 'loader-node-1'; ``` ## Related Links [Errors and Warnings](/errors-and-warnings) # Data Manipulation Language (DML) Statement Reference Source: https://docs.ocient.com/data-manipulation-language-dml-statement-reference Reference Data Manipulation Language (DML) statements in Ocient to insert, delete, truncate, and manage transactional changes to data in tables. DML statements let you add, remove, and manage the data stored in tables. You can write and execute DML statements like any other SQL statements in the System. The SELECT SQL statement is sometimes considered a DML statement, and the Ocient System includes it in the Data Query Language (DQL). For details, see the [Data Query Language (DQL) Statement Reference](/data-query-language-dql-statement-reference). Supported DML SQL statements are: * DELETE FROM TABLE * INSERT INTO TABLE * TRUNCATE TABLE ## DELETE FROM TABLE Removes rows from the specified table. You can use the `WHERE` clause to specify the rows to remove. If a `DELETE` SQL statement lacks the `WHERE` clause, then the database deletes all rows in the table. To use this statement, you must have the `DELETE` privilege for the table. For details and examples, see [Remove Records from an Ocient System](/remove-records-from-an-ocient-system). `DELETE` actions cannot be undone. If a `DELETE` operation fails during execution, the database rolls back the changes and returns to its original state. Due to limitations of the JDBC API, the reported modified row count might not be accurate for `DELETE` operations that are larger than two billion rows. **Syntax** ```sql SQL theme={null} DELETE FROM table_name [ WITH cte ] [ WHERE ] ``` | **Parameter** | **Type** | **Description** | | ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `table_name` | string | The name of the table, specified as a string, indicates where to delete rows. | | `cte` | string | A common table expression that defines temporary data for the `DELETE` statement.
For details about using common table expressions, see [WITH](/data-query-language-dql-statement-reference#with). | | `` | None | A logical combination of predicates that filter the rows to delete based on one or more columns.
For details, see the [WHERE](/data-query-language-dql-statement-reference#where) clause.
The `DELETE` SQL statement removes all rows from a table if you do not include the `WHERE` clause. | **Examples** **Delete Rows from the Table with Filter Criteria** This `DELETE` SQL statement removes all rows in the `movies` table that have a budget of less than `10000`. ```sql SQL theme={null} DELETE FROM movies WHERE budget < 10000; ``` **Delete Rows from the Table Using a Common Table Expression** This example uses a common table expression using the `WITH` keyword to find rows representing all transactions that occurred before 2022 that are less than \$100. The `DELETE` SQL statement receives the results from the common table expression. Then, the database executes this statement to delete the corresponding rows. ```sql SQL theme={null} DELETE FROM transactions WITH old_transactions AS ( SELECT transaction_id FROM transactions WHERE transaction_date < '2022-01-01' AND amount < 100 ) WHERE transaction_id IN ( SELECT transaction_id FROM old_transactions ); ``` ## INSERT INTO TABLE `INSERT INTO` inserts rows into a table in the current database using literal values, column references, function executions, computed expressions, or column default values. This SQL statement requires the `INSERT` privilege for the relevant table. Due to limitations of the JDBC API, the reported modified row count might not be accurate for insert operations that are larger than two billion rows. **Syntax** ```sql SQL theme={null} INSERT INTO table_name [ ( col1, col2 [, ...] ) ] [ WITH cte ] { query | [ DEFAULT VALUES | VALUES [ ] } ::= ( row1_exp1, row1_exp2 [, ...] ), ( row2_exp1, row2_exp2 [, ...] ) [, ...] ``` Using `DEFAULT VALUES` inserts a single row where each target column is populated with its column defaults (as defined in the column definition) instead of an explicit `VALUES` list. For table columns that do not each have a defined default value, the inserted row is NULL. If the column has no default and also has the `NOT NULL` constraint, the insert operation generates an error. | **Parameter** | **Type** | **Description** | | ------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `table_name` | string | The name of the table for insertion. | | `col1, col2 [, ... ]` | string | A list of specific columns to insert specific values.

This column list defaults to all columns in the table if you do not specify any column names.

The `INSERT` statement can use a subset of the table columns. Any columns not included in the statement are populated with their column default value in their column definition. If the column definition does not specify a default, the inserted row is NULL. If the column has no default and also has the `NOT NULL` constraint, the insert operation generates an error. | | `cte` | string | A common table expression that defines temporary data for the `INSERT` statement.
For details about using common table expressions, see [WITH](/data-query-language-dql-statement-reference). | | `query` | string | A `SELECT` query that defines values or a table and any of its columns that should be inserted into the specified `table_name`. | | `row1_exp1, row1_exp2 [, ...]` | string | The expressions to insert into columns in the table. This list must match the number of columns specified in the INSERT statement. Similarly, each expression must match the data type of the column that corresponds to its position.

Expressions can be any of the following:

**Literals**: `1`, `3.14`, '`abc`', `DATE '2024-01-01'`, etc.

**Column references:** If your `INSERT` statement includes a common table expression using a `WITH` clause, you can reference columns from the separate table in that clause.

**Function executions**: `ABS(-5)`, `NOW()`, `ST_DISTANCE`, etc.

**Computed expressions**: `price * quantity`, `COALESCE(x, 0)`, etc.

**Column default values**: Use the keyword `DEFAULT` to insert a default value specified in the column definition. If the column definition does not specify a default, the inserted row is NULL. If the column has no default and also has the `NOT NULL` constraint, the insert operation generates an error. | **Examples** **Insert Values from One Column** This example inserts the columns from `system.table_b` into `system.table_a`. ```sql SQL theme={null} INSERT INTO system.table_a SELECT * FROM system.table_b; ``` **Insert Values from Multiple Columns** This example inserts the column `system.table_b.id_col_b` into `system.table_a.id_col_a` and `system.table_b.int_col_b` into `system.table_a.int_col_a`. ```sql SQL theme={null} INSERT INTO system.table_a (id_col_a, int_col_a) SELECT id_col_b, int_col_b FROM system.table_b; ``` **Insert Literal Values** Create a table with product, quantity, and date of sale information with these non-nullable columns: * `product` — Product identifier * `quantity` — Quantity of the product sold * `sale_date` — Date of sale ```sql SQL theme={null} CREATE TABLE sales ( product INT NOT NULL, quantity INT NOT NULL, sale_date DATE NOT NULL ); ``` Insert three rows of literal values that represent different sales. ```sql SQL theme={null} INSERT INTO sales (product, quantity, sale_date) VALUES (1, 10, '2023-01-15'), (2, 5, '2023-01-20'), (1, 8, '2023-02-05'); ``` **Insert Values Using a Common Table Expression** In this example, a common table expression performs calculations on the `sales` table before inserting rows into the `monthly_sales_summary` table. The example uses the `monthly_sales_summary` table created by this `CREATE TABLE` statement with these non-nullable columns: * `product_id` — Product identifier * `month` — Month part of the date * `total_quantity` — Total quantity of the product ```sql SQL theme={null} CREATE TABLE monthly_sales_summary ( "product_id" INT NOT NULL, "month" DATE NOT NULL, "total_quantity" INT NOT NULL ); ``` The common table expression following the `WITH` keyword extracts the month from the sale date `sale_date` and calculates the sum of the quantity sold `total_quantity` of the product from the `sales` table before inserting this data. Then, the `INSERT` SQL statement specifies to insert the data into the `monthly_sales_summary` table. ```sql SQL theme={null} INSERT INTO monthly_sales_summary (product_id, month, total_quantity) WITH monthly_totals AS ( SELECT product, DATE_TRUNC('month', sale_date) AS month, SUM(quantity_sold) AS total_quantity FROM sales GROUP BY product, DATE_TRUNC('month', sale_date) ) SELECT product, month, total_quantity FROM monthly_totals; ``` **Insert Columns Using Default Values** This code utilizes the `customers` table with these columns: * `customer_id` — Customer identifier * `name` — Customer name * `status` — Customer status with the default `active` status * `created_at` — Created date ```sql SQL theme={null} CREATE TABLE customers ( customer_id INT, name VARCHAR(100), status VARCHAR(20) DEFAULT 'ACTIVE', created_at TIMESTAMP ); ``` Use the `DEFAULT VALUES` keyword to insert one row of default values into the table. For columns that lack a defined default value, the operation inserts a NULL row. ```sql SQL theme={null} INSERT INTO customers DEFAULT VALUES; ``` The resulting row contains all NULL values except for the `status` column, which has the `active` default value. ```sql SQL theme={null} SELECT * FROM customers; ``` Output ```sql SQL theme={null} | customer_id | name | status | created_at | | ----------- | ----- | ------ | ---------- | | | | ACTIVE | | ``` Alternatively, you can insert default values by using the `DEFAULT` keyword as one of the row values in the `INSERT` statement. ```sql SQL theme={null} INSERT INTO customers (customer_id, name, status, created_at) VALUES (1, 'Alice', DEFAULT, NULL); ``` Output ```sql SQL theme={null} | customer_id | name | status | created_at | | ----------- | ----- | ------ | ---------- | | | | ACTIVE | | | 1 | Alice | ACTIVE | | ``` ### INSERT INTO TABLE USING LOADERS Specify one or more Loader Nodes for executing the `INSERT INTO` SQL statement. If you do not use this option, the Ocient System uses all Loader Nodes that are live to execute the SQL statement. This statement is useful for managing loading operations, particularly when balancing multiple loads of different sizes and resource requirements. Alternatively, this statement can also help simplify small batch loads by sourcing the data from a single Loader Node. **Syntax** ```sql SQL theme={null} INSERT INTO TABLE table_name USING LOADERS streamloader [, ... ] query ``` | **Parameter** | **Type** | **Description** | | -------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `streamloader` | string | A unique name for the Loader Node.
Identify the names of Loader Nodes from the `sys.nodes` table by using this query: `SELECT name FROM sys.nodes;`
If the name of the streamloader contains special characters, you must enclose it in quotes, such as `"stream-loader1"`. | For the query to execute successfully, the specified names must: * Identify nodes that are live. * Identify nodes that have the Loader role. **Examples** This example inserts the column `system.table_b.id_col_b` into `system.table_a.id_col_a` and `system.table_b.int_col_b` into `system.table_a.int_col_a`. Use the Loader Node named `stream-loader1` to execute this SQL statement. ```sql SQL theme={null} INSERT INTO system.table_a (id_col_a, int_col_a) USING LOADERS "stream-loader1" SELECT id_col_b, int_col_b FROM system.table_b; ``` In this example, execute the same SQL statement with two Loader Nodes named `stream-loader2` and `stream-loader3`. ```sql SQL theme={null} INSERT INTO system.table_a (id_col_a, int_col_a) USING LOADERS "stream-loader2","stream-loader3" SELECT id_col_b, int_col_b FROM system.table_b; ``` ## TRUNCATE TABLE `TRUNCATE TABLE` removes some or all records from an existing table in the current database. The system deletes the truncated data, but the table and its schema remain intact in the system even if all data is deleted. If the entire table is truncated, Global Dictionary Compression tables remain in place. To truncate a table, you must have the `DELETE` privilege for the table. To remove a subset of rows from a table, you can use the [DELETE FROM TABLE](#delete-from-table) SQL statement. For details and examples of using `TRUNCATE`, see [Remove Records from an Ocient System](/remove-records-from-an-ocient-system). This action cannot be undone and results in data loss. **Syntax** ```sql SQL theme={null} TRUNCATE TABLE table_name TRUNCATE TABLE table_name WHERE segment_group_id = TRUNCATE TABLE table_name WHERE segment_group_id in (, ...) ``` | **Parameter** | **Type** | **Description** | | ------------------ | -------- | ---------------------------------- | | `table_name` | string | The name of the table to truncate. | | `segment_group_id` | numeric | Identifier of the segment group. | **Examples** This example truncates an existing table in the current database and schema named `students`. ```sql SQL theme={null} TRUNCATE TABLE students; ``` This example truncates an existing table in the current database named `us.students`. ```sql SQL theme={null} TRUNCATE TABLE us.students; ``` This example truncates a single segment group from an existing table in the current database named `students`. ```sql SQL theme={null} TRUNCATE TABLE students WHERE segment_group_id = 123456789; ``` This example truncates a number of segment groups from an existing table in the current database named `us.students`. ```sql SQL theme={null} TRUNCATE TABLE us.students WHERE segment_group_id IN (1,2,3,4,5); ``` ## Related Links [Data Query Language (DQL) Statement Reference](/data-query-language-dql-statement-reference) [Data Definition Language (DDL) Statement Reference](/data-definition-language-ddl-statement-reference) [Transactional Control Language (TCL) Statement Reference](/transaction-control-language-tcl-statement-reference) [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program) [Ocient Python Module (pyocient)](/ocient-python-module-pyocient) # Data Pipeline Behavior Considerations Source: https://docs.ocient.com/data-pipeline-behavior-considerations Understand how Ocient data pipelines handle file resumption, Kafka offsets, exactly-once semantics, deduplication, and database and table dependencies. Data pipelines enable you to load data from your chosen source into the . For details about the syntax, see [Data Pipelines](/data-pipelines). When you use data pipelines to load data, consider how to resume a pipeline with file loading, restart the pipeline with loading, and review pipeline dependencies. ### Resume a Pipeline with File Loading In many cases, a file-based pipeline stops executing before completion. You cannot resume a pipeline in a `COMPLETED` status. To resume a pipeline, use the `START PIPELINE` SQL statement. Before you resume a pipeline, the status of the pipeline must be `CREATED`, `FAILED`, or `STOPPED`. When a pipeline resumes, individual files remain in their most recent status as defined in the `sys.pipeline_files` system catalog table. For `BATCH` pipelines, the System does not add new files to the eligible file list when the pipeline resumes. If you modify the contents of files during the loading process, the Ocient System might experience issues with deduplication that cause duplicated rows or missing data. Avoid modifying files after you start a pipeline for the first time. Creating new files on your data source does not impact deduplication logic. The `START` operation groups files using their `extractor_task_id` and `stream_source_id` identifiers. The `stream_source_id` uniquely identifies partitions (i.e., an ordered list of files), and `extractor_task_id` identifies the batch that loads a group of partitions. #### File Statuses The Ocient System considers files with the statuses `LOADED`, `LOADED_WITH_ERRORS`, or `SKIPPED` to be in the terminal status, whereas other file statuses are still in process. * **Completed Batches** — If all the files in a particular batch have terminal status, then the pipeline does not attempt to reload the batch. These files have been completely processed, so the Ocient System ignores modifications to these files. * **In-Process Batches** — If at least one file in a particular batch does not have terminal status, then the pipeline reloads the entire batch. The pipeline reprocesses the in-process batches and relies on row deduplication to prevent duplication of rows in the target tables. * Modifications to files in an in-process batch can but are not guaranteed to be picked up by a restart. * Modifications to any files in this batch with the `LOADED`, `LOADED_WITH_ERRORS`, or `SKIPPED` statuses might cause issues with deduplication, leading to duplicate or missed data. * **Pending Files** — The Ocient System does not assign all `PENDING` files to a partition. The pipeline attempts to load these files after reloading any in-progress batches. #### Load Duplicate Data from Files Sometimes you might want to load the same data multiple times. If you want to load a second copy of the source data, you can either: * Drop and recreate the pipeline to reset the `sys.pipeline_files` system catalog table. * Create a second pipeline with a new name and the same configuration. When you truncate the target tables and restart the pipeline, the Ocient System does not reload the data. ### Restart with Kafka Loading Ocient relies on the offset management and consumer group behavior in Kafka to deliver exactly-once loading semantics and to control the Ocient pipeline behavior. #### Kafka Offsets and Consumer Group Identifiers If you set the `WRITE_OFFSETS` option to `true` (default value is `true`), the Kafka consumers commit offsets back to Kafka after data is considered durable in the database. The Kafka Broker stores these offsets as the last committed offset for the group identifier `group.id`. For each pipeline, the group identifier defaults to `____`, where the `` is the identifier of your system, `` is the name of your database, and `` is the name of the data pipeline. In most use cases, you should not manually change the `group.id` field for a pipeline. Any Kafka pipeline that has the same `group.id` starts consuming from its last committed offset, or if you do not set the value, the pipeline uses the Kafka `auto.offset.reset` policy to determine where to start. For details, see [Kafka offset management](https://docs.confluent.io/platform/current/clients/consumer.html#offset-management). If you want to start loading from the beginning of a topic, configure an unused `group.id` field (or use a `group.id` field that did not commit any of its offsets back) and ensure the `auto.offset.reset` Kafka configuration is appropriately set in the `CONFIG` option. #### Kafka Pipeline Deduplication The committed offset of a Kafka partition lags slightly behind the rows that have been loaded into Ocient. These lags do not cause an issue with data duplication. If you stop a pipeline before it can commit its most recent durable offset to Kafka, restarting the same pipeline starts loading from the last committed offset. However, the database deduplicates records sent twice for the same pipeline. Ocient deduplicates Kafka data for the specified combination of pipeline identifier, Kafka topic, and the Kafka partition number. While the consumer group offsets manage where the pipeline resumes loading, the Ocient System enforces the exactly-once loading of a Kafka partition only if you stop or restart a pipeline with the same pipeline identifier. If you drop a pipeline and create a new one with the same name, the Ocient System creates a new pipeline identifier. This action does not deduplicate data against data loaded in the original pipeline. To preserve deduplication, instead of dropping the pipeline, use the `CREATE OR REPLACE PIPELINE` SQL statement with the original pipeline name and the pipeline correctly deduplicates against the original data. Do not run multiple pipelines concurrently with the same consumer group identifier. This action leads to unpredictable data duplication. If you want to increase the number of consumers that read from a Kafka topic, increase the value of the `CORES` parameter. #### Load Duplicate Data on Kafka Sometimes you might want to load the same data multiple times. If you want to load a second copy of the source data from Kafka, you can either: * Drop the pipeline, recreate it with the same name, and reset the consumer group offsets manually. * Create a new pipeline with a different name and load from the beginning of the topic. ### Pipeline Database Dependency Each pipeline belongs to a database. You cannot drop a database that has a running pipeline. To drop a database, ensure that all pipelines in the database are in a non-running status. ### Pipeline Table Dependency Each pipeline has a target table. You cannot drop a table that has a running pipeline. To drop a table, ensure that all pipelines that are loading data into the table are in a non-running status. ## Related Links [Load Data](/load-data) [Data Pipeline Load of CSV Data from S3](/data-pipeline-load-of-csv-data-from-s3) [Data Pipeline Load of Parquet Data from S3](/data-pipeline-load-of-parquet-data-from-s3) [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) [Load Metadata and File-Based Partitioned Data in Data Pipelines](/load-metadata-and-file-based-partitioned-data-in-data-pipelines) [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference) [Identifiers](/identifiers) # Data Pipeline Load of CSV Data from S3 Source: https://docs.ocient.com/data-pipeline-load-of-csv-data-from-s3 Load CSV data from AWS S3 into Ocient using a data pipeline. This tutorial covers database, table, and pipeline creation, execution, and monitoring. A common setup for loading files in a batch into is to load from a bucket on S3 with time-partitioned data. Often, you must perform a batch load repeatedly to load new files. Ocient uses data pipelines to transform each document into rows in one or more different tables. The loading and transformation capabilities use a simple SQL-like syntax for transforming data. This tutorial guides you through a simple example load using a small data set in CSV format. The data in this example comes from a test set for the Business Intelligence tool. ## Prerequisites This tutorial assumes that: 1. The Ocient System has network access to S3 from the Loader Nodes. 2. An Ocient System is installed and configured with an active Storage Cluster (See the [Ocient Application Configuration](/ocient-application-configuration) guide). ## Step 1: Create a New Database Connect to a SQL Node using the [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). Then, execute the `CREATE DATABASE` SQL statement for a database named `metabase`. ```sql SQL theme={null} CREATE DATABASE metabase; ``` ## Step 2: Create Table Create the `orders` table in the new database. First, connect to that database (e.g., `connect to jdbc:ocient://sql-node:4050/metabase`), and then execute this `CREATE TABLE` SQL statement that specifies to create a table with these columns and a clustering index based on the `user_id` and `product_id` columns: * `created_at` as a timestamp that is not nullable. * `id`, `user_id`, and `product_id` as integers that are not nullable. * `subtotal`, `tax`, `total`, and `discount` as floating point numbers. * `quantity` as an integer. ```sql SQL theme={null} CREATE TABLE public.orders ( created_at TIMESTAMP TIME KEY BUCKET(30, DAY) NOT NULL, id INT NOT NULL, user_id INT NOT NULL, product_id INT NOT NULL, subtotal DOUBLE, tax DOUBLE, total DOUBLE, discount DOUBLE, quantity INT, CLUSTERING INDEX idx01 (user_id, product_id) ); ``` The database creates the `orders` table, and you can begin loading data. ## Step 3: Create a Data Pipeline Create data pipelines using the `CREATE PIPELINE` SQL statement. To load data, you first create a pipeline with the definition of the source, data format, and transformation rules using a SQL-like declarative syntax. Then, you execute the `START PIPELINE` SQL statement to start the load. You can observe progress and status using system tables and views. Each Ocient pipeline defines a single data source and the target table or tables into which data loads. A data source includes the location of the source and filters on the source to define the specific data set to load. This example loads data from a data source located in a directory within the S3 bucket. First, inspect the data that you plan to load. Each document has a format similar to this example CSV file named `orders.csv`. ```none Text theme={null} id,user_id,product_id,subtotal,tax,total,discount,created_at,quantity 1,1,14,37.65,2.07,39.72,null,2019-02-11T21:40:27.892Z,2 2,1,123,110.93,6.1,117.03,null,2018-05-15T08:04:04.580Z,3 3,1,105,52.72,2.9,49.2,6.42,2019-12-06T22:22:48.544Z,2 ... ``` In this case, Ocient automatically transforms the data to the target columns using some sensible conventions. In other cases, loads require some transformation. Most transformations are identical to functions that already exist in the SQL dialect of the Ocient System. Create a pipeline named `orders_pipeline` for the orders data set from your database connection prompt. Use the S3 data source with endpoint `https://s3.us-east-1.amazonaws.com`, bucket `ocient-docs`, and filter `metabase_samples/csv/orders.csv`. Specify the CSV format with one header line. Load the data into the `public.orders` table. The `SELECT` part of the SQL statement maps the fields in the CSV file to the target columns in the created table. ```sql SQL theme={null} CREATE PIPELINE orders_pipeline SOURCE S3 ENDPOINT 'https://s3.us-east-1.amazonaws.com' BUCKET 'ocient-docs' FILTER 'metabase_samples/csv/orders.csv' EXTRACT FORMAT csv NUM_HEADER_LINES 1 INTO public.orders SELECT $1 as id, $2 as user_id, $3 as product_id, $4 as subtotal, $5 as tax, $6 as total, $7 as discount, $8 as created_at, $9 as quantity; ``` The pipeline has three main sections: * `SOURCE` — In this case, load data from S3. Specify the endpoint and bucket. The `FILTER` parameter identifies the file or files to load. The load uses a single CSV file. Options exist to add wildcards or regular expressions to isolate different file sets. * `EXTRACT` — Set the format to CSV files and note that there is one header line in the file. This specification skips that row when the Ocient System processes the file. Many other options exist for delimited data such as a record delimiter and field delimiter. * `INTO ... SELECT` — Choose the target table `public.orders` and select the fields from the CSV file. The numeric index identifies each file field. Importantly, similar to other SQL syntax, the first field in the file is `$1`, not `$0`. Each field maps to a target column using the `as` syntax. After you successfully create the `orders_pipeline` pipeline, execute the `START PIPELINE` SQL statement. ```sql SQL theme={null} START PIPELINE orders_pipeline; ``` ## Step 4: Observe the Load Progress With your pipeline running, data immediately begins to load from the S3 files that you defined. If there are many files in each file group, the load process first sorts the files into batches, partitions them for parallel processing, and assigns them to Loader Nodes. You can check the pipeline status and progress by querying `information_schema.pipeline_status` or by executing `SHOW PIPELINE_STATUS` ```sql SQL theme={null} SHOW PIPELINE_STATUS; ``` *Output* ```sql SQL theme={null} database_name pipeline_name table_names status status_message percent_complete duration_seconds files_processed files_failed files_remaining records_processed records_loaded records_failed -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- metabase orders_pipeline ["public.orders"] RUNNING Started processing pipeline orders_pipeline 0.0 2.025824 0 0 1 0 0 0 ``` After the status of the pipeline changes to `COMPLETED`, all data is available in the target table. After a few seconds, the data is available for query in the `public.orders` table. ```sql SQL theme={null} SELECT COUNT(*) FROM public.orders; ``` *Output* ```sql SQL theme={null} count(*) -------------------- 18760 ``` You can drop the pipeline with the `DROP PIPELINE orders_pipeline;` SQL statement. Execution of this statement leaves the data in your target table, but removes metadata about the pipeline execution from the system. ## Related Links [Data Pipelines Reference](/data-pipelines) [Load Delimited and CSV Data](/data-formats-for-data-pipelines#load-delimited-and-csv-data) [Data Types for Data Pipelines](/data-types-for-data-pipelines) [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) [Manage Errors in Data Pipelines](/manage-errors-in-data-pipelines) # Data Pipeline Load of JSON Data from HDFS Source: https://docs.ocient.com/data-pipeline-load-of-json-data-from-hdfs Load JSON data from HDFS into the Ocient System using a data pipeline. Preview the data load, execute the pipeline, and monitor its progress. The data pipeline functionality enables the loading of data from . You can load various file types stored in HDFS into the System. The Ocient System uses data pipelines to transform each document into rows in one or more different tables. The loading and transformation capabilities use a simple SQL-like syntax for transforming data. This tutorial guides you through a simple example of loading data in the JSON format. ## HDFS Advanced Source Options Besides the specified options in the [HDFS Source Options](/data-pipelines#hdfs-source-options) section, you can also specify these options that are available for file system sources: * COMPRESSION\_METHOD * START\_FILENAME * END\_FILENAME * START\_CREATED\_TIMESTAMP * END\_CREATED\_TIMESTAMP * START\_MODIFIED\_TIMESTAMP * END\_MODIFIED\_TIMESTAMP * SORT\_BY * SORT\_DIRECTION * SORT\_REWRITE For the `CONFIG` source option, the most useful properties are the ones with the `dfs.client` prefix. For the full property reference, see [HDFS Default](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml) and [Core Default](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.html). Only properties that affect client connections and read operations apply to data pipeline loading. This table describes some common properties. | **Property Name** | **Property Description** | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `dfs.client.use.datanode.hostname` | Set this property to `true` when you have datanodes and namenodes in the HDFS cluster configured to connect to each other using local IP addresses, but loading occurs externally and requires externally valid hostnames instead. The default value is `false`. | | `dfs.client.retry.max.attempts` | Configure to change the number of times the loading process retries network requests to namenodes before throwing an error. You can increase this value when network connections are unstable. The default value is 10. | | `dfs.client.socket-timeout` | Configure to change the timeout (in milliseconds) for all sockets used internally in HDFS loading. You can increase this value when network connections are unstable. The default value is 60000 (60 seconds). | ## HDFS Loading Example Follow these steps to load JSON data from an HDFS source into the Ocient System. ### Step 1: Create a New Database Connect to a SQL Node using the [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). Then, execute the `CREATE DATABASE` SQL statement to create the `geo` database. ```sql SQL theme={null} CREATE DATABASE geo; ``` ### Step 2: Create a New Table in the Database Create the `locations` table in the `public` schema to store location data with a name, zip code, and a point with latitude and longitude: * `name` — Location name as a not nullable string * `zipcode` — Zip code as an integer * `location` — Location latitude and longitude as a point ```sql SQL theme={null} CREATE TABLE public.locations ( name VARCHAR(255) NOT NULL, zipcode INT NOT NULL, location POINT ); ``` ### Step 3: Preview and Create a Data Pipeline Preview the `locations` data pipeline to load JSON data from HDFS. This data pipeline uses the HDFS endpoint `hdfs-namenode:9000`, which consists of the namenode and port number, and the `/locations/2026/**/*.json` filter to load data from JSON files. The pipeline selects the name, zip code, and point data. For the point data construction, see [ST\_POINT](/point-constructors#st_point). ```sql SQL theme={null} PREVIEW PIPELINE locations SOURCE hdfs ENDPOINT 'hdfs-namenode:9000' FILTER '/locations/2026/**/*.json' EXTRACT FORMAT json INTO locations SELECT $name AS name, $zipcode AS zipcode, ST_POINT($longitude, $latitude) AS location; ``` Create the `locations` data pipeline to load JSON data from HDFS. ```sql SQL theme={null} CREATE PIPELINE locations SOURCE hdfs ENDPOINT 'hdfs-namenode:9000' FILTER '/locations/2026/**/*.json' EXTRACT FORMAT json INTO locations SELECT $name AS name, $zipcode AS zipcode, ST_POINT($longitude, $latitude) AS location; ``` After you successfully create the `locations` pipeline, execute the `START PIPELINE` SQL statement to start the data pipeline. ```sql SQL theme={null} START PIPELINE orders_pipeline; ``` ### Step 4: Observe the Load Progress With your pipeline running, data begins to load immediately from the JSON files. If there are many files in each file group, the load process first sorts the files into batches, partitions them for parallel processing, and assigns them to Loader Nodes. You can check the pipeline status and progress by querying the `information_schema.pipeline_status` system catalog table or executing the `SHOW PIPELINE_STATUS` SQL statement. ```sql SQL theme={null} SHOW PIPELINE_STATUS; ``` Output ```sql SQL theme={null} database_name pipeline_name table_names status status_message percent_complete duration_seconds files_processed files_failed files_remaining records_processed records_loaded records_failed -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- geo locations ["public.locations"] RUNNING Started processing pipeline locations 0.0 2.025824 0 0 1 0 0 0 ``` After the status of the pipeline changes to `COMPLETED`, all data is available in the target table. After a few seconds, the data is available for query in the `public.locations`table. ```sql SQL theme={null} SELECT COUNT(*) FROM public.locations; ``` *Output* ```sql SQL theme={null} count(*) -------------------- 25605 ``` You can drop the pipeline with the `DROP PIPELINE locations;` SQL statement. Execution of this statement leaves the data in your target table, but removes metadata about the pipeline execution from the system. ## Related Links [Data Pipelines Reference](/data-pipelines) [Load JSON Data](/data-formats-for-data-pipelines#load-json-data) [Data Types for Data Pipelines](/data-types-for-data-pipelines) [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) [Manage Errors in Data Pipelines](/manage-errors-in-data-pipelines) # Data Pipeline Load of JSON Data from Kafka Source: https://docs.ocient.com/data-pipeline-load-of-json-data-from-kafka Stream JSON data from Kafka topics into Ocient with data pipelines. Set up the database, table, pipeline, and SQL transformations for real-time analytics. A common setup for streaming data into is to load JSON data from an topic. Ocient uses data pipelines to transform each document into rows in one or more different tables. The loading and transformation capabilities use a simple SQL-like syntax for transforming data. This tutorial guides you through a simple example load using a small data set in JSON format. The data in this example comes from a test set for the Business Intelligence tool. ## Prerequisites This tutorial assumes that: 1. The Ocient System has network access to a Kafka broker from the SQL and Loader Nodes. 2. An Ocient System is installed and configured with an active Storage Cluster (see the [Ocient Application Configuration](/ocient-application-configuration) guide). ## Step 1: Create a New Database Connect to a SQL Node using the [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). Then, execute the `CREATE DATABASE` SQL statement to create the `metabase` database. ```sql SQL theme={null} CREATE DATABASE metabase; ``` ## Step 2: Create Table Create the `orders` table in the new database. First, connect to that database (e.g., `connect to jdbc:ocient://sql-node:4050/metabase`), and then execute this `CREATE TABLE` SQL statement that specifies to create a table with these columns and a clustering index based on the `user_id` and `product_id` columns: * `created_at` as a timestamp that is not nullable. * `id`, `user_id`, and `product_id` as integers that are not nullable. * `subtotal`, `tax`, `total`, and `discount` as floating point numbers. * `quantity` as an integer. ```sql SQL theme={null} CREATE TABLE public.orders ( created_at TIMESTAMP TIME KEY BUCKET(30, DAY) NOT NULL, id INT NOT NULL, user_id INT NOT NULL, product_id INT NOT NULL, subtotal DOUBLE, tax DOUBLE, total DOUBLE, discount DOUBLE, quantity INT, CLUSTERING INDEX idx01 (user_id, product_id) ); ``` The database creates the `orders` table, and you can begin loading data. ## Step 3: Create a Data Pipeline Create data pipelines using the `CREATE PIPELINE` SQL statement. To load data, you first create a pipeline with the definition of the source, data format, and transformation rules using a SQL-like declarative syntax. Then, you execute the `START PIPELINE` command to start the load. You can observe progress and status using system tables and views. Each Ocient pipeline defines a single data source and the target table or tables into which data loads. A data source includes the location of the source and filters on the source to define the specific data set to load. This example loads data from two data sources, where each source is located in a directory on the same S3 bucket. First, inspect the data that you plan to load. Each document has a format similar to this example. ```json JSON theme={null} {"id": 1, "user_id": 1, "product_id": 14, "subtotal": 37.65, "tax": 2.07, "total": 39.72, "discount": null, "created_at": "2019-02-11T21:40:27.892Z", "quantity": 2} {"id": 2, "user_id": 1, "product_id": 123, "subtotal": 110.93, "tax": 6.1, "total": 117.03, "discount": null, "created_at": "2018-05-15T08:04:04.580Z", "quantity": 3} ``` In this case, Ocient automatically transforms the data to the target columns using some sensible conventions. In other cases, loads require some transformation. Most transformations are identical to functions that already exist in the SQL dialect of the Ocient System. Create a pipeline named `orders_pipeline` for the orders data set from your database connection prompt. Specify the Kafka source with broker address `192.168.0.1:9092` (replace this example Kafka broker address with your address) and topic `orders`. Load the data into the `public.orders` table. The `SELECT` part of the SQL statement maps the JSON fields to the target columns in the created table. ```sql SQL theme={null} CREATE PIPELINE orders_pipeline SOURCE KAFKA BOOTSTRAP_SERVERS '192.168.0.1:9092' TOPIC 'orders' EXTRACT FORMAT json INTO public.orders SELECT $id as id, $user_id as user_id, $product_id as product_id, $subtotal as subtotal, $tax as tax, $total as total, $discount as discount, $created_at as created_at, $quantity as quantity; ``` The pipeline has three main sections: * `SOURCE` — Loads data from Kafka. Specify the address of the bootstrap servers and topic. Options exist to set Kafka consumer configurations. * `EXTRACT` — Sets the format to JSON. * `INTO ... SELECT` — Targets the `public.orders` table and selects the chosen fields from the JSON records. In this case, all data is available at the top level of the JSON records, so the example references the fields by the attribute name (e.g., `$id`, `$user_id`, etc.). For nested data, reference the nested fields using dot notation (e.g., `$order.user.first_name`). Each field maps to a target column using the `as` syntax. After you successfully create this pipeline, execute the `START PIPELINE` SQL statement. ```sql SQL theme={null} START PIPELINE orders_pipeline; ``` ## Step 4: Confirm that Loading is Operating Correctly With the pipeline in place and running, data immediately begins loading off of the Kafka topics that are configured in the pipeline. If you do not have data in the Kafka topics yet, now is a good time to start producing data into the topics. ### Produce Test Data into Kafka For test purposes, [kafkacat](https://github.com/edenhill/kcat) is a helpful utility that makes it easy to produce records into a topic. For example, if you have a file of sample data `orders.jsonl` in a JSONL format (newline-delimited JSON records), you can run this command to send those records into your Kafka broker. Specify your broker IP address instead of `` and your topic name ``. ```shell Shell theme={null} kafkacat -b :9092 -t -T -P -l orders.jsonl ``` Save an example document to your file system to use for this test. For this example, you can download an example file from [https://ocient-docs.s3.amazonaws.com/metabase\_samples/jsonl/orders.jsonl](https://ocient-docs.s3.amazonaws.com/metabase_samples/jsonl/orders.jsonl) and save it to `~/orders.jsonl`. Assuming the broker is running at the IP address `10.0.0.3`, send data into the `orders` topic defined in the pipeline definition by executing this command. ```shell Shell theme={null} kafkacat -b 10.0.0.3:9092 -t orders -T -P -l orders.jsonl ``` This command pushes the entire JSONL file of messages into Kafka with one record per line. As these records are produced into Kafka, the running pipeline begins to load them into Ocient. ## Step 5: Observe the Load Progress With your pipeline running, data immediately begins to load from the Kafka topic. The pipeline creates parallel Kafka consumers for each partition. If there are more partitions than processing cores available, the pipeline automatically handles spreading consumers across available processing cores and Loader Nodes. You can check the pipeline status and progress by querying `information_schema.pipeline_status` or executing the `SHOW PIPELINE_STATUS` SQL statement. ```sql SQL theme={null} SHOW PIPELINE_STATUS; ``` *Output* ```sql SQL theme={null} database_name pipeline_name table_names status status_message percent_complete duration_seconds files_processed files_failed files_remaining records_processed records_loaded records_failed -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- metabase orders_pipeline ["public.orders"] RUNNING Started processing pipeline orders_pipeline 0.0 2.025824 0 0 1 0 0 0 ``` After a few seconds, the data is available for query in `public.orders`. ```sql SQL theme={null} SELECT COUNT(*) FROM public.orders; ``` *Output* ```sql SQL theme={null} count(*) -------------------- 18760 ``` Unlike a batch file load, Kafka pipelines run continuously, so they never change to a status of `COMPLETED`. To examine progress, you can use the information schema and system catalog tables. Key details are in `information_schema.pipelines`, `information_schema.pipeline_status`, and `sys.pipeline_partitions`. For example, this statement shows the status and key metrics for duration, loaded records, and failed records. ```sql SQL theme={null} SELECT pipeline_name, status, status_message, duration_seconds, records_loaded, records_failed FROM information_schema.pipeline_status; ``` *Output* ```sql SQL theme={null} pipeline_name status status_message duration_seconds records_loaded records_failed -------------------------------------------------------------------------------------------------------------------------------- orders_pipeline RUNNING Started processing pipeline orders_pipeline 10.089936 18760 0 ``` In this example, there are no errors. The `sys.pipeline_errors` system catalog table captures any errors that occur during the pipeline process. You can drop the pipeline with the `DROP PIPELINE orders_pipeline;` SQL statement. Execution of this statement leaves the data in your target table, but removes metadata about the pipeline execution from the system. ## Related Links [Data Pipelines Reference](/data-pipelines) [Load JSON Data](/data-formats-for-data-pipelines#load-json-data) [Data Types for Data Pipelines](/data-types-for-data-pipelines) [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) [Manage Errors in Data Pipelines](/manage-errors-in-data-pipelines) # Data Pipeline Load of Parquet Data from S3 Source: https://docs.ocient.com/data-pipeline-load-of-parquet-data-from-s3 Load time-partitioned Parquet files from AWS S3 into Ocient tables with data pipelines and SQL transformations. Includes setup, execution, and monitoring. A common setup for loading files in a batch into the System is to load from a bucket on S3 with time-partitioned data. Often, you must perform a batch load repeatedly to load new files. The Ocient System uses data pipelines to transform each document into rows in one or more different tables. The loading and transformation capabilities use a simple SQL-like syntax for transforming data. This tutorial guides you through a simple example load using a small data set in format. The data in this example comes from a test set for the Business Intelligence tool. ## Parquet Loading Recommendations Follow this set of recommendations for an optimal loading experience of Parquet files. **File Configuration** * Files should have row groups of less than 128 MB. Larger row groups can impact memory usage during loading, and row groups of 512 MB can cause loading failures on 1 TB or more data sets. * Encoding fields in a Parquet file reduces the space of the file on disk but can impact memory usage during loading. Enable encoding on fields that you expect to have less than 256 unique values and for fields that contain short strings. You do not have to encode other fields. **Multiple Files** * You can load row groups of multiple Parquet files in parallel. For large data sets, load the data set as multiple files. * Loading files with differing schemas is not supported. ## Prerequisites This tutorial assumes that: 1. The Ocient System has network access to S3 from the Loader Nodes. 2. An Ocient System is installed and configured with an active Storage Cluster. For details, see [Ocient Application Configuration](/ocient-application-configuration). ## Parquet Loading Example Follow these steps to load Parquet data into the Ocient System. ### Step 1: Create a New Database Connect to a SQL Node using the [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). Then, execute the `CREATE DATABASE` SQL statement to create the `metabase` database. ```sql SQL theme={null} CREATE DATABASE metabase; ``` ### Step 2: Create a New Table in the Database Create the `orders` table in the new database. First, connect to that database (e.g., `connect to jdbc:ocient://sql-node:4050/metabase`), and then execute this `CREATE TABLE` SQL statement that specifies to create a table with these columns and a clustering index based on the `user_id` and `product_id` columns: * `created_at` as a timestamp that is not nullable. * `id`, `user_id`, and `product_id` as integers that are not nullable. * `subtotal`, `tax`, `total`, and `discount` as floating point numbers. * `quantity` as an integer. ```sql SQL theme={null} CREATE TABLE public.orders ( created_at TIMESTAMP TIME KEY BUCKET(30, DAY) NOT NULL, id INT NOT NULL, user_id INT NOT NULL, product_id INT NOT NULL, subtotal DOUBLE, tax DOUBLE, total DOUBLE, discount DOUBLE, quantity INT, CLUSTERING INDEX idx01 (user_id, product_id) ); ``` The database creates the `orders` table, and you can begin loading data. ### Step 3: Preview and Create a Data Pipeline Create data pipelines using the `CREATE PIPELINE` SQL statement. To load data, you first create a pipeline with the definition of the source, data format, and transformation rules using a SQL-like declarative syntax. Then, you execute the `START PIPELINE` SQL statement to start the load. You can observe progress and status using system catalog tables and views. Each Ocient pipeline defines a single data source and the target table or tables into which data loads. A data source includes the location of the source and filters on the source to define the specific data set to load. This tutorial loads data from two data sources, where each source is located in a directory on the same S3 bucket. First, inspect the data that you plan to load. Each document has a JSON format similar to this example. ```json JSON theme={null} {"id": 1, "user_id": 1, "product_id": 14, "subtotal": 37.65, "tax": 2.07, "total": 39.72, "discount": null, "created_at": "2019-02-11T21:40:27.892Z", "quantity": 2} {"id": 2, "user_id": 1, "product_id": 123, "subtotal": 110.93, "tax": 6.1, "total": 117.03, "discount": null, "created_at": "2018-05-15T08:04:04.580Z", "quantity": 3} ``` Inspecting the file with Pandas, you can see this schema with details. ```none Parquet Schema Description theme={null} # Column Non-Null Count Dtype --- ------ -------------- ----- 0 id 18760 non-null int64 1 user_id 18760 non-null int64 2 product_id 18760 non-null int64 3 subtotal 18760 non-null float64 4 tax 18760 non-null float64 5 total 18760 non-null float64 6 discount 1915 non-null float64 7 created_at 18760 non-null datetime64[ns, UTC] 8 quantity 18760 non-null int64 ``` In this case, Ocient automatically transforms the data to the target columns using some sensible conventions. In other cases, loads require some transformation. Most transformations are identical to functions that already exist in the SQL syntax of the Ocient System. Prior to creating a pipeline, you can use the `PREVIEW PIPELINE` SQL statement to create your pipeline iteratively. This statement returns a result set that shows the final values that would be loaded but does not load the data into the target table. Preview the `orders_pipeline` pipeline for the `orders` data set from your database connection prompt. Use the S3 data source with endpoint `https://s3.us-east-1.amazonaws.com`, bucket `ocient-docs`, and filter `metabase_samples/parquet/orders.parquet`. Specify the `parquet` format. Load the data into the `public.orders` table. The `SELECT` part of the SQL statement maps the fields in the Parquet file to the target columns in the created table. In this example, limit the result set to the first five records in the data source. ```sql SQL theme={null} PREVIEW PIPELINE orders_pipeline SOURCE S3 ENDPOINT 'https://s3.us-east-1.amazonaws.com' BUCKET 'ocient-docs' FILTER 'metabase_samples/parquet/orders.parquet' LIMIT 5 EXTRACT FORMAT parquet INTO public.orders SELECT $id as id, $user_id as user_id, $product_id as product_id, $subtotal as subtotal, $tax as tax, $total as total, $discount as discount, $created_at as created_at, $quantity as quantity; ``` *Output* ```sql SQL theme={null} id user_id product_id subtotal tax total discount created_at quantity ----------------------------------------------------------------------------------------------------------------------------------------------------------------- 1 1 14 37.65 2.07 39.72 NULL 2019-02-11 21:40:27.892000000 2 8 1 123 110.93 6.1 117.03 NULL 2018-05-15 08:04:04.580000000 3 8 1 105 52.72 2.9 49.2 6.42 2019-12-06 22:22:48.544000000 2 8 1 94 109.22 6.01 115.23 NULL 2019-08-22 16:30:42.392000000 6 8 1 132 127.88 7.03 134.91 NULL 2018-10-10 03:34:47.309000000 5 Fetched 5 rows ``` With this preview, you can confirm that the results of the pipeline match your requirements. If there is an issue, you can update the statement and run the `PREVIEW PIPELINE` statement again until it meets your needs. Next, create the pipeline named `orders_pipeline` for the `orders` data set from your database connection prompt. The pipeline has three main sections: * `SOURCE` — Loads data from S3. Set the S3 endpoint, bucket name, and filter for the Parquet files. * `EXTRACT` — Sets the format to Parquet. * `INTO ... SELECT` — Targets the `public.orders` table and selects the chosen fields from the Parquet records. In this case, all data is available at the top level of the Parquet records, so the example references the fields by the attribute name (e.g., `$id`, `$user_id`, etc.). For nested data, reference the nested fields using dot notation (e.g., `$order.user.first_name`). Each field maps to a target column using the `as` syntax. ```sql SQL theme={null} CREATE PIPELINE orders_pipeline SOURCE S3 ENDPOINT 'https://s3.us-east-1.amazonaws.com' BUCKET 'ocient-docs' FILTER 'metabase_samples/parquet/orders.parquet' EXTRACT FORMAT parquet INTO public.orders SELECT $id as id, $user_id as user_id, $product_id as product_id, $subtotal as subtotal, $tax as tax, $total as total, $discount as discount, $created_at as created_at, $quantity as quantity; ``` After you successfully create this pipeline, execute the `START PIPELINE` SQL statement to start the load. ```sql SQL theme={null} START PIPELINE orders_pipeline; ``` ### Step 4: Observe the Load Progress With your pipeline running, data begins to load immediately from the S3 files that you defined. If there are many files in each file group, the load process first sorts the files into batches, partitions them for parallel processing, and assigns them to Loader Nodes. You can check the pipeline status and progress by querying the `information_schema.pipeline_status` system catalog table or executing the `SHOW PIPELINE_STATUS` SQL statement. ```sql SQL theme={null} SHOW PIPELINE_STATUS; ``` *Output* ```sql SQL theme={null} database_name pipeline_name table_names status status_message percent_complete duration_seconds files_processed files_failed files_remaining records_processed records_loaded records_failed -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- metabase orders_pipeline ["public.orders"] RUNNING Started processing pipeline orders_pipeline 0.0 2.025824 0 0 1 0 0 0 ``` After the status of the pipeline changes to `COMPLETED`, all data is available in the target table. After a few seconds, the data is available for query in the `public.orders` table. ```sql SQL theme={null} SELECT COUNT(*) FROM public.orders; ``` *Output* ```sql SQL theme={null} count(*) -------------------- 18760 ``` You can drop the pipeline with the `DROP PIPELINE orders_pipeline;` SQL statement. Execution of this statement leaves the data in your target table, but removes metadata about the pipeline execution from the system. ## Related Links [Data Pipelines Reference](/data-pipelines) [Load JSON Data](/data-formats-for-data-pipelines#load-json-data) [Data Types for Data Pipelines](/data-types-for-data-pipelines) [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) [Manage Errors in Data Pipelines](/manage-errors-in-data-pipelines) # Data Pipeline Loading Errors Source: https://docs.ocient.com/data-pipeline-loading-errors Reference of Ocient data pipeline error codes and messages, with parameter descriptions and recommended actions for resolving load and parsing failures. This table captures error codes and their corresponding error messages that you can encounter when you load data using data pipelines in the System. Also, the table contains the definitions of the parameters in the error messages, descriptions that explain the error messages, and actions you can take to resolve the errors. | **Code** | **Error Message** | **Message Parameters** | **Description** | **Resolution** | | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | OLD00 | Key `key` for class `class` must be set to a non-NULL value. Details: `details` | `key`: The key of the missing property.
`class`: The class that has the missing property.
`details`: Additional information about the error. | This error occurs when the loading process reads the pipeline definition missing a required property. | Specify a non-NULL value for the specified key in the pipeline definition. | | OLD01 | Value `value` is not acceptable with the key `key` for class `class`. Expected `example`. Details: `details` | `value`: The value that you set for the specified pipeline definition key.
`key`: The pipeline definition key with an incorrect value.
`class`: The class that contains the property.
`example`: An example or description of the correct value.
`details`: Additional information about the error. | This error occurs when the loading process reads the pipeline definition that contains a key with an incorrect value. | Specify the correct value for the setting corresponding to the specified key in the pipeline definition. | | OLD02 | No class with name `class` exists in the pipeline builder packages. Pipeline cannot run. | `class`: The class that cannot be found. | This error occurs when the loading process reads the pipeline definition that contains an unknown class. | Ensure that the class exists and belongs to a package specified in the `'engine.builder.packages'` configuration option. | | OLE00 | Failed to extract record with index `index` from `sourceName`. Details: `details` | `index`: The index of the record (index starts at 1) in the source where the error occurred.
`sourceName`: The name of the source resource where the error occurred.
`details`: Additional information about the error. | This error occurs when the extraction of a record fails. | Provide the correct source data in the accurate format, extract configuration, or compression method. | | OLE01 | `dataExtract` cannot be extracted from `sourceData`. Details: `details` | `dataExtract`: The data to extract.
`sourceData`: The source data for the extraction.
`details`: Additional information about the error | This error occurs when the extraction of data from a record fails. | Provide the correct source data in the accurate format, extract configuration, compression method, or transform configuration. | | OLE02 | The source data has encountered invalid data or character sets. If invalid data is expected in the load, turn off the VALIDATE\_CHARACTERS option or use the REPLACE\_INVALID\_CHARACTERS option instead. | None | This error occurs when the pipeline has the VALIDATE\_CHARACTERS option turned on and the load encounters invalid data.
| If you want to skip files that contain invalid data, turn on the TOLERATE file error mode. If you expect invalid data in the load, turn off the VALIDATE\_CHARACTERS option. You can replace invalid data with the REPLACE\_INVALID\_CHARACTERS option. | | OLM00 | The
`message`
is too large.
`modification` | `message`: The type of message that is too large.

`modification`: An example or description of a modification to limit the payload size of the message. | This error occurs when an internal communication fails because the message size exceeds the maximum size set by the system.
| Modify the corresponding command to limit the payload size of the message.
| | OLO00 | Failed to connect to `sourceType` source because the specified `locationType` `location` was not found. Details: `details` | `sourceType`: The type of the source.
`locationType`: The type of the data location argument that was not found.
`location`: The value of the data location argument that was not found.
`details`: Additional information about the error. | This error occurs when the pipeline cannot find the source data (e.g., cannot find the configured bucket or file path). | In the pipeline definition, specify an existing data location in the source. | | OLO01 | Failed to connect to `sourceType` source because the access credentials are unauthorized. Details: `details` | `sourceType`: The type of source where access is denied.
`details`: Additional information about the error. | This error occurs when the pipeline cannot connect to the source due to invalid credentials. | Specify credentials in the pipeline definition with access to the specified source or configure the source to provide access to the pipeline. | | OLO10 | File list is empty at the `sourceType` source provided. The locations searched are: `location`.
Details: `details` | `sourceType`: The type of the source.
`location`: The value of the data location argument that was not found. | This error occurs when the pipeline cannot find the source data (e.g., the file path is empty). | If an empty file list was expected, set the `expect.empty.file.list` configuration option to `true` by executing `ALTER SYSTEM ALTER CONFIG SET 'streamloader.extractorEngineParameters.configurationOption.expect.empty.file.list' = 'true';`. Then, restart the pipeline. | | OLO50 | File `fileName` was not found. | `fileName`: The name of the missing file. | This error occurs when the pipeline is configured to fail on missing files, and it cannot find a file in the list. | Handle the missing file using your own process or configure the pipeline to skip missing files instead. | | OLS00 | Failed to load transformed value `value` to sink after `transformName`. Details: `details` | `value`: The value that the Loader Node could not load to the sink.
`transformName`: The name of the executed transform.
`details`: Additional information about the error. | This error occurs when the Loader Node cannot load a transformed value to the sink. For example, the transformed value might be too large for the sink column. | Check that the sink column is configured correctly or check that the transforms in the column are composed properly to return values that can load to the sink. | | OLS97 | No table with name `tableName` exists in sink at `ipAddress`:`port`. The available tables are: `existingTables`. | `tableName`: The fully specified name of the specified table.
`ipAddress`: The IP address for the Loader Node of the Ocient sink.
`port`: The external TCP port for the Loader Node of the Ocient sink.
`existingTables`: The list of existing tables in the same database and schema as the specified table. | This error occurs when no table with the specified name exists in the sink, but the database and schema of the table do exist. | Specify an existing table in the pipeline definition or create a table with the specified name. | | OLS98 | The column name `columnName` does not exist. The columns available to the pipeline from the `tableName` table are: `existingColumns`. | `columnName`: The name of the column that was not found.
`tableName`: The table that was searched.
`existingColumns`: The columns in the table that are available to the pipeline. | This error occurs when the target table in the sink does not contain all the columns specified in the pipeline or a column specified as a transform target is not declared in the sink definition of the pipeline.
(The database ignores deleted columns. This error does not occur when you load data to a deleted column.) | Specify an existing column in the pipeline definition or add the corresponding column to the target table. | | OLS99 | Failed to connect to sink at `ipAddress`:`port`. Check that a streamloader at that address is reachable. | `ipAddress`: The IP address for the Loader Node of the Ocient sink.
`port`: The external TCP port for the Loader Node of the Ocient sink. | This error occurs when the pipeline cannot connect to its Ocient sink using a Loader Node. | Check that the Loader Node at the specified host and port is reachable and that the host and port of the Ocient sink are configured properly. | | OLT00 | Cannot perform operation `operationName` because the inputs are `inputType`. Expected inputs are `expectedType`. | `operationName`: The name of the operation that cannot be performed because its inputs are the wrong type. (usually the name of a transform)
`inputType`: The type of inputs that the operation received.
`expectedType`: The type of inputs that the operation expects to receive. | This error occurs when transforms in a column are not composed correctly. | Add, remove, or replace transforms such that each transform receives input of the correct type and that the Loader Node loads data of the correct type into the target column. | | OLT01 | Failed to transform value `value` using `transformName`. Details: `details` | `value`: The value that was not transformed.
`transformName`: The name of the failed transform.
`details`: Additional information about the error. | This error occurs when a transformation fails. | Correct the source data or modify the transform to fit the source data. Alternatively, start the pipeline with a higher error limit so the load skips the failing record. | | OLT02 | Value `value` is not an acceptable `parameterName` for `transformName`. Expected `example`. | `value`: The incorrect value for the specified parameter.
`parameterName`: The name of the specified parameter.
`transformName`: The name of the transform that is not configured correctly.
`example`: An example or description of the correct value. | This error occurs when the parameter of the transform is not configured correctly. | Correct the parameter of the transform. | | OLT03 | Failed to transform value `value` using `transformName` because the value is `valueDescription`. Expected `example`. | `value`: The value that was not transformed.
`transformName`: The name of the failed transform.
`valueDescription`: A feature or description of the incorrect value for this transform.
`example`: An example or description of the correct value. | This error occurs when a transformation fails because the data is not correct for the transform. | Correct the source data, modify the transform configuration, or use a different transform. | | OLT04 | Failed to match value `value` to format `format` in `transformName` because `valuePart` at position `valuePartPosition` does not match pattern `pattern`. Expected `example`. | `value`: The value that was not parsed with the transform format.
`format`: The transform format.
`transformName`: The name of the failed transform.
`valuePart`: The part of the value that was unmatched.
`valuePartPosition`: The position (index starts at 1) of the part unmatched in the value.
`pattern`: The pattern in the transform format that did not match part of the value.
`example`: An example or description of the correct value. | This error occurs when a transformation fails because the value does not match the specified transform format. | Correct the source data or modify the transform format to fit the source data. | | OLTM0 | Failed to transform `value` value using `transformName`. `details`. | `value`: The value that was not transformed.
`transformName`: The name of the failed transform.
`details`: Additional information about the error. | This error occurs when an EXPLODE transformation fails. | Correct the source data or modify the transform to fit the source data. Alternatively, start the pipeline with a higher error limit so the load skips the failing record. | | OL998 | An error occurred in library `library`: `details` | `library`: The name of the library where the error occurred.
`details`: Additional information about the error. | This error occurs when the Ocient System detects an error from an internal library. | Diagnose and address the error using the details provided by the library. | | OL999 | An error occurred in class `class`: `details` | `class`: The class that threw the exception.
`details`: Additional information about the error. | This error occurs when the Ocient System detects a general error. | If the details of the message are not helpful for you to resolve the error, then contact Ocient Support. | ## Related Links [Errors and Warnings](/errors-and-warnings) [Monitor Data Pipelines](/monitor-data-pipelines) # Data Pipelines Source: https://docs.ocient.com/data-pipelines Overview of Ocient data pipelines, the SQL-based ETL framework with CREATE PIPELINE DDL commands for ingesting and transforming streaming and batch data. Data pipelines enable you to load data from your chosen source into the . You can preview a data pipeline using the `PREVIEW PIPELINE` SQL statement to test the creation statement and any required data transformation. Then, you can create a data pipeline using the `CREATE PIPELINE` statement. To manage the load, use the `START PIPELINE` statement to start the load and the `STOP PIPELINE` statement to stop it. You can evolve the schema of a data pipeline using the `ALTER PIPELINE` statement. You can also rename a data pipeline using the `ALTER PIPELINE RENAME` statement. To see the full definition of a created pipeline, use the `EXPORT PIPELINE` statement. When you finish with the load, you can use the `DROP PIPELINE` statement to remove the data pipeline. You can create user-defined data pipeline functions using the `CREATE PIPELINE FUNCTION` statement and remove the function using the `DROP PIPELINE FUNCTION` statement. Also, you can administer privileges for data pipelines and data pipeline functions. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). ## CREATE PIPELINE `CREATE PIPELINE` defines a data pipeline that you can execute with the `START PIPELINE` SQL statement. Specify the type of load, data source, and data format. You must have the `ALTER` privilege on the pipeline. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). **Syntax** ```sql SQL theme={null} CREATE [ OR REPLACE ] [ BATCH | CONTINUOUS | TRANSACTIONAL ] PIPELINE [ IF NOT EXISTS ] pipeline_name [ ] [ BAD_DATA_TARGET ] SOURCE ( | | | ) [ MONITOR ( | ) ] [ LOOKUP lookup_source ] [ ... ] EXTRACT ( | | | | | | ) [ ] [ INSERT ] INTO destination_table_name SELECT expression as col_alias_target, expression2 as col_alias_target2, ... [ WHERE filter_expression ] [ [ INSERT ] INTO destination_table_name_n SELECT expression_n as col_alias_target_n, expression2_n as col_alias_target2_n, ... [ WHERE filter_expression_n ] ] [ ... ] /********************/ /* ADVANCED OPTIONS */ /********************/ advanced_pipeline_options ::= [ CORES processing_cores ] [ PARTITIONS file_partitions ] [ BATCH_SIZE number_of_rows ] [ RECORD_NUMBER_FORMAT record_number_format ] /***************************/ /* BAD_DATA_TARGET OPTIONS */ /***************************/ kafka_bad_data_target ::= KAFKA BOOTSTRAP_SERVERS bootstrap_servers TOPIC topic_name [ CONFIG config_option ] /******************/ /* SOURCE OPTIONS */ /******************/ s3_source ::= S3 BUCKET bucket_name ( FILTER | FILTER_GLOB | FILTER_REGEX | OBJECT_KEY ) specifiers [ REGION region ] [ ENDPOINT endpoint ] [ ENABLE_PATH_STYLE_ACCESS enable_path_style_access ] [ ACCESS_KEY_ID access_key_credentials ] [ SECRET_ACCESS_KEY secret_key_credentials ] [ SESSION_TOKEN session_token ] [ ROLE_ARN role_arn ] [ ASSUME_ROLE_CONFIG assume_role_config] [ MAX_CONCURRENCY parallel_connections ] [ READ_TIMEOUT num_seconds ] [ REQUEST_DEPTH num_requests ] [ REQUEST_RETRIES num_retries ] [ HEADERS headers ] filesystem_source ::= FILESYSTEM ( FILTER | FILTER_GLOB | FILTER_REGEX ) specifiers hdfs_source ::= HDFS ( FILTER | FILTER_GLOB | FILTER_REGEX ) specifiers ENDPOINT endpoint [ CONFIG hdfs_config ] kafka_source ::= KAFKA BOOTSTRAP_SERVERS bootstrap_servers TOPIC topic_name [ WRITE_OFFSETS write_offsets ] [ CONFIG config_option ] [ AUTO_OFFSET_RESET ( 'latest' | 'earliest' ) file_based_source_options ::= [ PREFIX prefix ] [ COMPRESSION_METHOD 'gzip' ] [ SORT_BY ( 'filename' | 'created' | 'modified' ) [ SORT_DIRECTION ( ASC | DESC ) ] ] [ SORT_REWRITE sort_rewrite ] [ START_FILENAME start_filename ] [ END_FILENAME end_filename ] [ START_CREATED_TIMESTAMP start_created_timestamp ] [ END_CREATED_TIMESTAMP end_created_timestamp ] [ START_MODIFIED_TIMESTAMP start_modified_timestamp ] [ END_MODIFIED_TIMESTAMP end_modified_timestamp ] /******************/ /* MONITOR OPTIONS */ /******************/ sqs_monitor ::= SQS QUEUE_URL queue_url [ REGION region ] [ ENDPOINT endpoint ] [ ACCESS_KEY_ID access_key_id ] [ SECRET_ACCESS_KEY secret_access_key ] kafka_monitor ::= KAFKA BOOTSTRAP_SERVERS bootstrap_servers TOPIC topic AUTO_OFFSET_RESET ( 'latest' | 'earliest' ) [ GROUP_ID group_id ] [ MESSAGE { FILENAME file_selector TIMESTAMP timestamp_selector SIZE size_selector } ] [ CONSUMER { MAX_MESSAGES max_messages TIMEOUT consumer_timeout } ] general_monitor_options ::= [ POLLING_INTERVAL polling_interval ] [ BATCH { MAX_FILES max_files TIMEOUT batch_timeout LOOKBACK lookback } ] /*******************/ /* LOOKUP OPTIONS */ /*******************/ lookup_options ::= CONNECTION_TYPE connection_type CONNECTION_STRING connection_string LOOKUP_SCHEMA lookup_schema LOOKUP_TABLE lookup_table [ REFRESH_PERIOD refresh_period ] [ CONFIG config_json ] /*******************/ /* EXTRACT OPTIONS */ /*******************/ general_extract_options ::= [ CHARSET_NAME charset_name ] [ COLUMN_DEFAULT_IF_NULL column_default_if_null ] [ NULL_STRINGS null_strings ] [ TRIM_WHITESPACE trim_whitespace ] [ VALIDATE_CHARACTERS validate_characters ] [ REPLACE_INVALID_CHARACTERS replace_invalid_characters ] [ REPLACEMENT_CHARACTER replacement_character ] delimited_extract_options ::= FORMAT ( 'delimited' | 'csv' ) [ COMMENT_CHAR comment_char ] [ EMPTY_FIELD_AS_NULL empty_field_as_null ] [ ESCAPE_CHAR escape_char ] [ FIELD_DELIMITER field_delimiter ] [ FIELD_OPTIONALLY_ENCLOSED_BY enclosure_char ] [ HEADERS delimited_headers ] [ NUM_HEADER_LINES num_header_lines ] [ NUM_FOOTER_LINES num_footer_lines ] [ RECORD_DELIMITER record_delimiter ] [ SKIP_EMPTY_LINES skip_empty_lines ] [ OPEN_ARRAY open_array_char ] [ CLOSE_ARRAY close_array_char ] [ ARRAY_ELEMENT_DELIMITER array_delimiter ] [ OPEN_OBJECT open_object_char ] [ CLOSE_OBJECT close_object_char ] [ STRIP_ARRAY_ELEMENT_QUOTES strip_array_element_quotes ] [ STRIP_FIELD_QUOTES strip_field_quotes ] [ TRIM_ARRAY_ELEMENTS trim_array_elements] asn1_extract_options ::= FORMAT 'asn.1' SCHEMA { URL url_asn1 RECORD_TYPE record_type } avro_extract_options ::= FORMAT 'avro' [ SCHEMA { [ INLINE inline_string ] [ INFER_FROM 'sample_file' ] [ URL url_avro ] [ CONFIG config_avro ] [ SUBJECT subject ] [ SCHEMA_REGISTRY_ID_LOCATION ( 'header_or_value' | 'value' | 'none' ) ] } ] binary_extract_options ::= FORMAT 'binary' RECORD_LENGTH length_in_bytes [ ENDIANNESS ( 'big' | 'little' ) ] [ AUTO_TRIM_PADDING auto_trim_padding ] [ PADDING_CHARACTER padding_character ] json_extract_options ::= FORMAT 'json' parquet_extract_options ::= FORMAT 'parquet' SCHEMA { INFER_FROM infer_from } xml_extract_options ::= FORMAT 'xml' ``` ### Pipeline Identity and Naming The name of a data pipeline is unique in an System. Reference the pipeline in other SQL statements like `START PIPELINE`, `STOP PIPELINE`, and `DROP PIPELINE` using the name of the pipeline. The SQL statement throws an error if a pipeline with the same name already exists unless you specify the `IF NOT EXISTS` option. You can rename a pipeline with the `ALTER PIPELINE RENAME` SQL statement. ### Update a Pipeline You can update a pipeline with the `OR REPLACE` clause in the `CREATE PIPELINE` SQL statement. Use this clause when you want to continue loading from the current place in a continuous load, but you need to modify transformations or other settings. If you specify the `OR REPLACE` clause and the pipeline already exists, the database replaces the original pipeline object with the options specified in the new `CREATE OR REPLACE PIPELINE` statement. When you replace an existing pipeline, the pipeline retains its current position in the source data so that data is not duplicated when the pipeline is resumed with a `START PIPELINE` SQL statement. First, you must stop a pipeline before executing the `CREATE OR REPLACE` SQL statement. ### Data Pipeline Modes You can define pipelines in either batch `BATCH`, continuous `CONTINUOUS`, or transactional `TRANSACTIONAL` mode. * File sources (e.g., `s3`, `filesystem`) support `BATCH` and `CONTINUOUS` modes. File-based loads default to `BATCH` mode if you do not specify this keyword. * only supports `CONTINUOUS` mode. Loads with a Kafka source default to `CONTINUOUS` mode if you do not specify this keyword. * When you execute the `START PIPELINE` SQL statement using a data pipeline in the `BATCH` mode, the system creates a static list of files with the `PENDING` status in the `sys.pipeline_files` system catalog table. With the `CONTINUOUS` mode, the monitor appends new incoming files to the list of files in the `sys.pipeline_files` system catalog table. * With the `CONTINUOUS` mode, you can apply filters for the data pipeline. The system only uses files in the consumed messages if the filenames match the filters. For this mode, these options are invalid: * PREFIX * SORT\_BY * SORT\_DIRECTION * SORT\_REWRITE * START\_CREATED\_TIMESTAMP * END\_CREATED\_TIMESTAMP * START\_MODIFIED\_TIMESTAMP * END\_MODIFIED\_TIMESTAMP * The `TRANSACTIONAL` mode enables the data pipeline to roll back the interim results of the load if an error occurs during load execution. The system makes all data visible only after the load completes successfully. This mode supports all options that the `BATCH` mode does. For details, see [Transactional Data Pipelines](/transactional-data-pipelines). ### Required Options You must define certain options in every pipeline. For these options, there is no default value. Optional options need to be present only if you use specific functions. A pipeline must contain the `SOURCE`, `EXTRACT FORMAT`, and `INTO table_name ... SELECT` statements. * [SOURCE Options](#source-options) * [EXTRACT Options](#extract-options) * [Data Formats for Data Pipelines](/data-formats-for-data-pipelines) * [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) These options depend on each other: `IF NOT EXISTS` and `OR REPLACE` SQL statements are mutually exclusive. ### SELECT Statement and Data Transformation Use the `INTO table_name ... SELECT` SQL statement in the `CREATE PIPELINE` SQL statement to specify how to transform the data. The `SELECT` statement includes a set of one or more expressions and a target column name in the form `expression as column_name`. The `expression` part in the statement contains a source field reference (e.g., `$1` or `$my_field.subfield`) and, optionally, the transformation function you want to apply. If your data is nested in arrays, you can use special transformation functions, such as the [EXPLODE\_OUTER](/special-data-pipeline-transformation-functions) function, to expand the data into individual rows. For details about data transformation and supported transformation functions, see [Transform Data in Data Pipelines](/transform-data-in-data-pipelines). For details about data types and casting, see [Data Types for Data Pipelines](/data-types-for-data-pipelines) and [Data Formats for Data Pipelines](/data-formats-for-data-pipelines). You can also specify metadata values, such as filename, to load in the `SELECT` SQL statement. For details, see [Load Metadata and File-Based Partitioned Data in Data Pipelines](/load-metadata-and-file-based-partitioned-data-in-data-pipelines). This is an example of a transformation statement snippet. ```sql SQL theme={null} ... FORMAT json INTO public.orders SELECT TIMESTAMP(BIGINT($created_timestamp)) as ordertime, metadata('filename') as source_filename, $order_number as ordernumber, $customer.first_name as fname, LEFT($customer.middle_initial,1) as minitial, $customer.last_name as lname, $postal_code as postal_code, $promo_code as promo_code, $order_total as ordertotal, DECIMAL($tax,8,2) as tax, CHAR[]($line_items[].product_name) as product_names, CHAR[]($line_items[].sku) as skus ``` Optionally, you can filter the load by using the `WHERE` clause in the form `WHERE filter_expression`. This expression should evaluate to a Boolean value or NULL and can include more than one filter expression. The system loads rows that contain data matching the filter criteria when the expression evaluates to `true`. The system does not load the rows when the expression evaluates to `false` or NULL. You can include any transform in the `WHERE` clause as in the `SELECT` clause. This is an example of a filter snippet that loads order data for customers whose last name starts with the letter `a`. Use the `COALESCE` function to return only non-NULL values. ```sql SQL theme={null} ... FORMAT json INTO public.orders SELECT TIMESTAMP(BIGINT($created_timestamp)) as ordertime, metadata('filename') as source_filename, $order_number as ordernumber, $customer.first_name as fname, LEFT($customer.middle_initial,1) as minitial, $customer.last_name as lname, $postal_code as postal_code, $promo_code as promo_code, $order_total as ordertotal, DECIMAL($tax,8,2) as tax, CHAR[]($line_items[].product_name) as product_names, CHAR[]($line_items[].sku) as skus WHERE STARTSWITH(COALESCE($customer.last_name, ''),'a') ``` #### NULL and Default Value Handling The Ocient System has specific ways to handle NULL and default values in the data pipeline. Consider this information when you prepare data to load and write the `CREATE TABLE` and `CREATE PIPELINE` SQL statements. To load NULL values, insert the NULL value in the data. In this case, if you specify `NOT NULL` for the target column in the `CREATE TABLE` statement, the data pipeline fails to load. If you omit a column in the `SELECT` SQL statement, the data pipeline loads the default value for the column. If you do not specify the default value, the pipeline loads a NULL value. If you specify `NOT NULL` for the target column, the data pipeline also fails to load. You can use the `DEFAULT` keyword to load a default value. In this case, if the column does not have a default value, the pipeline fails to load. If the pipeline loads a NULL into a column with a specified default value, you can use `COALESCE(, DEFAULT)` to insert the default value instead of the NULL value, where `` is the NULL column. You can modify the load of one column at a time in this way. This table describes the data pipeline behavior for NULL or omitted column values. | **Value in the Data Pipeline** | **Nullable Target Column** | **Default Value in Target Column** | **Resulting Data Pipeline Behavior** | | -------------------------------------------- | ------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | NULL | No | Value might or might not be set. | The pipeline uses the default value if the default exists and you specify the `COLUMN_DEFAULT_IF_NULL` option.
Otherwise, the pipeline fails. | | NULL | Yes | Value might or might not be set. | The pipeline uses the default value if the default exists and you specify the `COLUMN_DEFAULT_IF_NULL` option.
Otherwise, the pipeline uses the NULL value. | | Omitted column in the `SELECT` SQL statement | NULL value might or might not be set. | Yes | The pipeline uses the default value. | | Omitted column in the `SELECT` SQL statement | No | No | The pipeline fails. | | Omitted column in the `SELECT` SQL statement | Yes | No | The pipeline uses the NULL value. | ### Required Privileges You must have the `CREATE PIPELINE` privilege on the underlying database and the `VIEW` privilege on each table in the pipeline definition to execute the `CREATE PIPELINE` SQL statement. The table must already exist. See the `START PIPELINE` SQL statement for the required privileges to execute a pipeline. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). ### Examples #### Load JSON Data from Kafka This example loads JSON data from Kafka using the `CREATE PIPELINE` SQL statement. Use the bootstrap server `192.168.0.1:9092` using the Kafka topic `orders`. Load data into the `public.orders` table. Specify the data to load as these JSON selectors: * Identifier `$id` * User identifier `$user_id` * Product identifier `$product_id` * Subtotal amount `$subtotal` * Tax `$tax` * Total amount `$total` * Discount amount `$discount` * Created time `$created_at` * Quantity `$quantity` ```sql SQL theme={null} CREATE PIPELINE orders_pipeline SOURCE KAFKA BOOTSTRAP_SERVERS '192.168.0.1:9092' TOPIC 'orders' EXTRACT FORMAT json INTO public.orders SELECT $id as id, $user_id as user_id, $product_id as product_id, $subtotal as subtotal, $tax as tax, $total as total, $discount as discount, $created_at as created_at, $quantity as quantity; ``` For a complete tutorial, see [Data Pipeline Load of JSON Data from Kafka](/data-pipeline-load-of-json-data-from-kafka). #### Load Delimited Data from S3 This example loads delimited data in CSV format from S3. Use the [`https://s3.us-east-1.amazonaws.com`](https://s3.us-east-1.amazonaws.com) endpoint with the `ocient-docs` bucket and path `metabase_samples/csv/orders.csv`. Denote path-style access using the `ENABLE_PATH_STYLE_ACCESS` option set to `true`. Specify one header line with the `NUM_HEADER_LINES` option. Load data into the `public.orders` table. Specify the data to load using the column numbers: * Idenfier * User identifier * Product identifier * Subtotal amount * Tax * Total amount * Discount amount * Created time * Quantity ```sql SQL theme={null} CREATE PIPELINE orders_pipeline SOURCE S3 ENDPOINT 'https://s3.us-east-1.amazonaws.com' BUCKET 'ocient-docs' FILTER 'metabase_samples/csv/orders.csv' ENABLE_PATH_STYLE_ACCESS true EXTRACT FORMAT csv NUM_HEADER_LINES 1 INTO public.orders SELECT $1 as id, $2 as user_id, $3 as product_id, $4 as subtotal, $5 as tax, $6 as total, $7 as discount, $8 as created_at, $9 as quantity; ``` For a complete tutorial, see [Data Pipeline Load of CSV Data from S3](/data-pipeline-load-of-csv-data-from-s3). #### Continuous File Load of Delimited Data in CSV Files Create a data pipeline that uses continuous file loading with delimited data in CSV files. Specify to use 32 partitions and 16 cores with the `PARTITIONS` and `CORES` options, respectively. The source is S3 with the `http://endpoint.ocient.com` endpoint and `cs_data` bucket. Specify the filter `'*.csv'` to find all files with a filename that matches a glob pattern without subdirectories, for example, `data.csv`. The system filters filenames with subdirectories such as `data/data_sample.csv`. For continuous file loading, specify the `MONITOR` option to use the Kafka monitor. Specify the `test-broker:9092` bootstrap server, `cfl_kafka_ten_adtech_flat_small` Kafka topic, and reset the offset to the smallest offset using the `earliest` value of the `AUTO_OFFSET_RESET` option. Set the client group identifier to `84079bf1-bdc4-4b10-ba12-41ba6b17dffe`. The format is `csv` with the record delimiter as the newline character `\n`. Load data in the `public.ad_sessions` table. The `SELECT` statement identifies the columns to load by number. There are 39 fields in the CSV data. For each column, transform each column using cast functions. For details, see [Scalar Data Conversion Functions](/scalar-data-conversion-functions) for each function. See [Date and Time Functions](/date-and-time-functions) for the `TO_TIMESTAMP` function. ```sql SQL theme={null} CREATE CONTINUOUS PIPELINE test_continuous_pipeline PARTITIONS 32 CORES 16 SOURCE S3 ENDPOINT 'http://endpoint.ocient.com/' BUCKET 'cs_data' FILTER '*.csv' MONITOR kafka BOOTSTRAP_SERVERS 'test-broker:9092' TOPIC 'cfl_kafka_ten_adtech_flat_small' AUTO_OFFSET_RESET 'earliest' GROUP_ID '84079bf1-bdc4-4b10-ba12-41ba6b17dffe' EXTRACT FORMAT csv RECORD_DELIMITER '\n' INTO public.ad_sessions SELECT TO_TIMESTAMP(CHAR($1), 'yyyy-MM-dd HH:mm:ss.SSSSSS', 'java') AS event_date_time, CHAR($2) AS device_model, TINYINT($8) AS device_user_age, BOOLEAN($10) AS device_ad_tracking_disabled, BINARY($11) AS device_mac, INT($14) AS ip_zip, FLOAT($19) AS ip_zip_latitude, DOUBLE($20) AS ip_zip_longitude, BIGINT($21) AS session_id, SMALLINT($32) AS session_response_latency, DECIMAL($34, 10, 1) AS session_transaction_revenue, CHAR($39) AS session_app_name ; ``` #### Transactional Load of Delimited Data in CSV Files Create a data pipeline that uses transactional file loading with delimited data in CSV files. The source is S3 with the `http://endpoint.ocient.com` endpoint and `cs_data` bucket. Specify the filter `'*.csv'` to find all files with a filename that matches a glob pattern without subdirectories, for example, `data.csv`. The system filters filenames with subdirectories such as `data/data_sample.csv`. The format is `csv`. Load data in the `public.ad_sessions` table. The `SELECT` statement identifies the columns to load by number. For each column, transform each column using cast functions. For details, see [Scalar Data Conversion Functions](/scalar-data-conversion-functions) for each function. See [Date and Time Functions](/date-and-time-functions) for the `TO_TIMESTAMP` function. ```sql SQL theme={null} CREATE TRANSACTIONAL PIPELINE test_continuous_pipeline SOURCE S3 ENDPOINT 'http://endpoint.ocient.com/' BUCKET 'cs_data' FILTER '*.csv' EXTRACT FORMAT csv INTO public.ad_sessions SELECT TO_TIMESTAMP(CHAR($1), 'yyyy-MM-dd HH:mm:ss.SSSSSS', 'java') AS event_date_time, CHAR($2) AS device_model, TINYINT($8) AS device_user_age, BOOLEAN($10) AS device_ad_tracking_disabled, BINARY($11) AS device_mac, INT($14) AS ip_zip, FLOAT($19) AS ip_zip_latitude, DOUBLE($20) AS ip_zip_longitude, BIGINT($21) AS session_id, SMALLINT($32) AS session_response_latency, DECIMAL($34, 10, 1) AS session_transaction_revenue, CHAR($39) AS session_app_name ; ``` ### SOURCE Options #### File-Based Source Options Options that apply to both the `S3`, `FILESYSTEM`, and `HDFS` sources. | **Option Key** | **Default** | **Data Type** | **Description** | | --------------------------------------- | ----------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | FILTER \| FILTER\_GLOB \| FILTER\_REGEX | None | STRING | The expression for filtering files in the directory or child directories to load. If you do not specify the `PREFIX` option, this pattern applies to the full path to the file, except for the Bucket for S3-compatible file sources. If you specify the `PREFIX` option, this pattern applies to the file path portion after the specified prefix value.

Paths include a leading forward slash.

For `FILESYSTEM` sources, this option is required.

For `S3` sources, one of the `FILTER`, `FILTER_GLOB`, `FILTER_REGEX`, or `OBJECT_KEY` keys is required.

Specify one of these options:
\* `FILTER_GLOB` or `FILTER` — Regular filename patterns. Supported options include:
- `*` indicates unlimited wildcard characters except for the path character, `/`.
- `**` indicates wildcard characters, including path separators.
- `?` indicates a single wildcard character.
\* `FILTER_REGEX` — Regular expression patterns to filter files. Some common patterns include:
- `.` matches any character.
- `*` matches any 0 or more of the preceding character.
- `+` matches a single character.
- `[135]` matches any one character in the set.
- `[1-5]` matches any one character in the range.
- `(a\|b)` matches a or b.
ℹ️ When you use the FILTER or FILTER\_GLOB options with a continuous data pipeline, the system supports only basic globbing, and extended, range globbing is not supported.

**FILTER\_GLOB Examples:**
List all CSV files in subdirectories named `2024` that are in any subdirectory of the `trades` directory, which is in the root of the bucket.
`FILTER_GLOB = '/trades/*/2024/*.csv'`
List all CSV files in the bucket in all subdirectories.
`FILTER_GLOB = '**/*.csv'`

**FILTER\_REGEX Examples:**
List all `json.gz` files in the bucket that contain the name `orders` anywhere in the path.
`FILTER_REGEX = '.*orders.*\.json\.gz'`
List all files in the bucket in the root path `metrics` or `values` where the date string on the filename is 2000 to 2004.
`FILTER_REGEX = '/(metrics\|values)/.*200[0-4].*'` | | PREFIX | `NULL` | STRING
or
ARRAY OF STRINGS | Optional.

Specify a prefix within which to apply the filter. When you specify a list of prefixes, the system applies the filter to each one, and the results are unioned together.

Prefixes must be paths to directories, meaning they end with a forward slash. For `FILESYSTEM` sources, paths should be absolute, meaning they begin with a forward slash.

For `S3` sources, use prefixes to reduce the search space when listing objects within the specified `BUCKET`. For maximum performance, use the `PREFIX` option whenever possible, especially when combined with the `FILTER_REGEX` option.

ℹ️This option is not available for continuous data pipelines.

**Examples:**
List all CSV files in the `2024` subdirectory of `files`.
`PREFIX '/files/2024/' FILTER '**/*.csv'`
List all JSON files of subdirectory `2024/09/` in the `orders` directory.
`PREFIX '/data/orders/2024/09/' FILTER_REGEX '.*/orders/.*json'`
List all JSON files in multiple subdirectories.
`PREFIX ['/data/orders/2024/09/', '/data/orders/2024/10/']` | | COMPRESSION\_METHOD | `NULL` | STRING | Optional.

The method the Ocient System uses to decompress file data. The only supported option is `gzip`.
To load uncompressed data, omit the COMPRESSION\_METHOD option.

ℹ️The COMPRESSION\_METHOD option is only applicable to file-based sources. Kafka pipelines automatically decompress data based on the Kafka topic configuration. | | SORT\_BY | `filename` | STRING | Optional.

The sort criteria for sorting the file list before the load. Supported options are:
\* `filename` — Sort files lexicographically by the filename.
\* `created` — Sort files based on the file created timestamp.
\* `modified` — Sort files based on the file modified timestamp.
Sorting files such that the Ocient System sorts the data within the files according to the column in the target table can lead to better query performance.

ℹ️This option is not available for continuous data pipelines. | | SORT\_DIRECTION | `ASC` | STRING | Optional.

The sort direction, either `ASC` ascending or `DESC` descending, determines the sort order. If you specify this option, you must also specify the SORT\_BY option.

ℹ️This option is not available for continuous data pipelines. | | SORT\_REWRITE | `NULL` | STRING | Optional.

This option is a sort comparator, allowing files to be renamed during the sort operation using capture groups from the filter regular expression. If you specify this option, you must also specify the `FILTER_REGEX` option.

If you do not specify this option, the system uses the sorting specified by the SORT\_BY option.

ℹ️This option is not available for continuous data pipelines.

Example:

`FILTER_REGEX '(\d+)_(\d+)_(\d+)-order.json'`
`SORT_REWRITE '\3-\1-\2.json'`
This specification renames the file from `5_6_1999-orders.json` to `1999_5_6.json` by switching the order of the date fields. | | START\_FILENAME | `NULL` | STRING | Optional.

The filename string that is the lower bound to filter for files lexicographically for batch pipelines. (inclusive)
Use the full path of the file, such as `'/dir/load_file.json'`. If the file is located in the top-most directory, start the path with a slash `/`, such as `'/load_file.json'`.
You can use this option without the END\_FILENAME option.
If you specify the END\_FILENAME option, the value of the START\_FILENAME option must be smaller than the value for the END\_FILENAME option lexicographically. | | END\_FILENAME | `NULL` | STRING | Optional.

The filename string that is the upper bound to filter for files lexicographically for batch pipelines. (inclusive)
Use the full path of the file, such as `'/dir/load_file.json'`. If the file is located in the top-most directory, start the path with a slash `/`, such as `'/load_file.json'`.
You can use this option without the START\_FILENAME option.
If you specify the END\_FILENAME option, the value of the START\_FILENAME option must be smaller than the value for the END\_FILENAME option lexicographically. | | START\_CREATED\_TIMESTAMP | `NULL` | TIMESTAMP-formatted STRING | Optional.

The ISO-8601-compliant date or date time that is used as the lower bound (inclusive) to filter files by created timestamp for batch pipelines. The time zone should match the file metadata.

If you specify both, the value for the START\_CREATED\_TIMESTAMP option must be before the value for the END\_CREATED\_TIMESTAMP option.

ℹ️This option is not available for continuous data pipelines. | | END\_CREATED\_TIMESTAMP | `NULL` | TIMESTAMP-formatted STRING | Optional.

The ISO-8601-compliant date or date time that is used as the lower bound (inclusive) to filter files by created timestamp for batch pipelines. The time zone should match the file metadata.

If you specify both, the value for the START\_CREATED\_TIMESTAMP option must be before the value for the END\_CREATED\_TIMESTAMP option.

ℹ️This option is not available for continuous data pipelines. | | START\_MODIFIED\_TIMESTAMP | `NULL` | TIMESTAMP-formatted STRING | Optional.

The ISO-8601-compliant date or date time that is used as the lower bound (inclusive) to filter files by modified timestamp for batch pipelines. The time zone should match the file metadata.

If you specify both, the value for the START\_MODIFIED\_TIMESTAMP option must be before the value for the END\_MODIFIED\_TIMESTAMP option.

ℹ️This option is not available for continuous data pipelines. | | END\_MODIFIED\_TIMESTAMP | `NULL` | TIMESTAMP-formatted STRING | Optional.

The ISO-8601-compliant date or date time that is used as the lower bound (inclusive) to filter files by modified timestamp for batch pipelines. The time zone should match the file metadata.

If you specify both, the value for the START\_MODIFIED\_TIMESTAMP option must be before the value for the END\_MODIFIED\_TIMESTAMP option.

ℹ️This option is not available for continuous data pipelines. | #### S3 Source Options You can apply these options to data sources of the `SOURCE S3` type, which include S3 and S3-compatible services. | **Option Key** | **Default** | **Data Type** | **Description** | | --------------------------- | --------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | BUCKET | None | STRING | The name of the bucket in AWS S3. | | OBJECT\_KEY | `NULL` | STRING
or
ARRAY OF STRINGS | Optional.

Specify an S3 object key(s) to load. You cannot use this option if you specify `PREFIX`, `FILTER`, `FILTER_GLOB`, or `FILTER_REGEX` keys are specified. If you do not specify one of the `FILTER`, `FILTER_GLOB`, or `FILTER_REGEX` keys, this option is required.

Object keys do not include a leading forward slash. Specifying individual objects can be faster than filtering, because the Ocient System avoids listing all the files in a large directory.

Object keys must not include any of the special characters: `*?{}[]`

ℹ️This option is not available for continuous data pipelines.

**Examples:**
Load a single object from the designated bucket at the specified object key.
`OBJECT_KEY 'order_data/jsonl/orders_20251101.jsonl'`
Load a list of objects from the designated bucket at the specified object keys.
`OBJECT_KEY ['order_data/jsonl/orders_20251101.jsonl', 'order_data/jsonl/orders_20251201.jsonl']` | | ACCESS\_KEY\_ID | `''` | STRING | Optional.

The access key identification for AWS credentials. If you specify this option, you must also specify the `SECRET_ACCESS_KEY` option. The Ocient System uses anonymous credentials when you specify an empty value for this option. | | SECRET\_ACCESS\_KEY | `''` | STRING | Optional.

The secret key for AWS credentials. If you specify this option, you must also specify the ACCESS\_KEY\_ID option. The Ocient System uses anonymous credentials when you specify an empty value for this option. | | SESSION\_TOKEN | `NULL` | STRING | Optional.

Temporary credentials can be made with a combination with the existing access key identifier and secret key, and an additional session token. You must specify the ACCESS\_KEY\_ID and SECRET\_ACCESS\_KEY options when you use this option. | | ROLE\_ARN | `NULL` | STRING | Optional.

The Resource Name (ARN) of the role to retrieve temporary credentials. The STS client used to retrieve the temporary credentials specifies the [region](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-region-selection.html#automatically-determine-the-aws-region-from-the-environment) loaded from `DefaultAwsRegionProviderChain` and [credentials](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) loaded from `DefaultCredentialsProvider`. You cannot use this option with the ACCESS\_KEY\_ID and SECRET\_ACCESS\_KEY options. | | ASSUME\_ROLE\_CONFIG | `{}` | STRING | Optional.

Additional configuration used to configure the `StsAssumeRoleCredentialsProvider` or `AssumeRoleRequest` used to retrieve and refresh temporary credentials with the ROLE\_ARN option. This option is a JSON-formatted string that you can only specify at the same time as the ROLE\_ARN option. The supported configuration options are: `"asyncCredentialUpdateEnabled"`, `"prefetchTimeSeconds"`, `"staleTimeSeconds"` for `StsAssumeRoleCredentialsProvider` and `"durationSeconds"`, `"externalId"`, `"policy"`, `"roleSessionName"`, `"serialNumber"`, `"tokenCode"` for `AssumeRoleRequest`. | | REGION | `'us-east-1'` | STRING | Optional.

The region that the Ocient System uses for AWS access.
If you specify the `ENDPOINT` option, the system ignores this option. | | ENDPOINT | `NULL` | STRING | Optional.

The endpoint URI for the S3-compatible service API. (e.g., `https://s3.us-east-2.amazonaws.com`)

When unspecified, this option defaults to `https://s3.REGION.amazonaws.com`.

If you provide this option, the Ocient System ignores the `REGION` option. | | ENABLE\_PATH\_STYLE\_ACCESS | `NULL` | BOOLEAN | Optional.

Whether to use path-style access, where the path includes the bucket name in the URL. For example, `https://s3.us-east-1.amazonaws.com/bucket-name/key-name`. The `false` value denotes virtual-hosted style access, where the URL contains the bucket name as part of the domain name. For example, `https://bucket-name.s3.us-east-1.amazonaws.com/key-name`.

When unspecified, this option defaults to `true` if the `ENDPOINT` option is specified and `false` otherwise. | | HEADERS | `NULL` | STRING | Optional.

The headers to send with every request. This option is a JSON-formatted string. Represent the chosen header names as keys with corresponding values as scalars or lists of scalars. The system converts scalars that are not strings to strings. During the load, the system maps each element in a list to the header name represented by the corresponding key.

Examples:

`HEADERS '{"x-amz-request-payer": "requester"}'` returns this header for each request: `x-amz-request-payer: requester`.

`HEADERS '{"header-name": ["list", "of", "values"]}'` returns this header for each request: `header-name: list, of, values`. | | MAX\_CONCURRENCY | 50 | INTEGER | Optional.

Determines the number of parallel connections the Ocient System uses to communicate with the AWS S3 service.
⚠️ This option does not require modification in most cases. Contact Ocient Support to modify these values. | | READ\_TIMEOUT | 0 (unlimited timeout) | TIME INTERVAL | Optional.

The time until a read operation times out.
⚠️ This option does not require modification in most cases. Contact Ocient Support to modify these values. | | REQUEST\_DEPTH | 500 | INTEGER | Optional.

The upper boundary of requests that the Ocient System handles concurrently.
⚠️ This option does not require modification in most cases. Contact Ocient Support to modify these values. | | REQUEST\_RETRIES | 10 | INTEGER | Optional.

Number of times the AWS SDK retries failing requests before the Ocient System throws an error.
⚠️ This option does not require modification in most cases. Contact Ocient Support to modify these values. | **S3 Credentials Hierarchy** The S3 source configuration supports this hierarchy to obtain S3 credentials: * Level 1: Data pipeline configuration (set by using the `ACCESS_KEY_ID` and `SECRET_ACCESS_KEY` options) * Level 2: [AWS SDK Default Credential Provider Chain ](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html)(set by using the instructions provided in the web page) * Level 3: Anonymous access (set by default) If the Ocient System does not obtain the credentials at a level, the system tries a lower level. You can choose the level to store the credentials, where Level 1 is the highest. Higher levels take precedence for credential storage. #### FILESYSTEM Source Options No options exist specific to the file system source (`SOURCE FILESYSTEM`) except for general [file-based source options](#file-based-source-options). When you load data using `SOURCE FILESYSTEM`, the files must be addressable from all of your Loader Nodes. The Ocient System uses the specified path in the pipeline PREFIX and FILTER options to select the files to load. A shared view of the files you want to load must be available to all Loader Nodes involved in a pipeline. For example, you can use a Network File System (NFS) mount available to all Loader Nodes at a common path on each node. **Example** This example `CREATE PIPELINE` SQL statement snippet contains a `FILESYSTEM` source and filters to all CSV files in the `/tmp/sample-data/` directory on each of the Loader Nodes. ```sql SQL theme={null} CREATE PIPELINE ... SOURCE FILESYSTEM PREFIX '/tmp/sample-data/' FILTER '**/*.csv' EXTRACT FORMAT delimited INTO public.orders SELECT $1 as username, $2 as subtotal, ... ``` #### HDFS Source Options You can specify these options for data sources (`SOURCE HDFS`). For advanced options, see [Data Pipeline Load of JSON Data from HDFS](/data-pipeline-load-of-json-data-from-hdfs). | **Option Key** | **Default** | **Data Type** | **Description** | | -------------- | ----------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ENDPOINT | None | STRING | Required.

The hostname and port number of the namenode server. For example, `'my-hdfs-server:1234'`.

The namenode server manages the file system and regulates access to data. | | CONFIG | None | JSON-formatted STRING | Optional.

The JSON string that contains
HDFS [configuration properties](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml)
to use during the load. The properties are primarily for authentication. You can also use these properties to configure advanced properties such as connection timeouts, parallelism, and network chunk sizes.

**Example:**
`CONFIG '{"dfs.client.use.datanode.hostname": true, "dfs.client.socket-timeout": 300000}'` | #### KAFKA Source Options You can apply these options to Kafka data sources (`SOURCE KAFKA`). For compression, you do not need to specify a compression option in Kafka-based pipelines, because the Ocient System handles the compression type automatically. Records produced to the Kafka broker with a `compression.type` setting or with the `compression.type` set on the topic automatically decompress when the loading process consumes the records. The loading process uses built-in headers in Kafka to determine the required decompression during extraction. | **Option Key** | **Default** | **Data Type** | **Description** | | ------------------- | ---------------------------------------------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BOOTSTRAP\_SERVERS | None | STRING | A comma-delimited list of `IP:port` pairs that contain the IP addresses and the associated port numbers of the Kafka Brokers. You can also use a hostname instead of the IP address.
**Example:**
`BOOTSTRAP_SERVERS = '198.51.100.1:9092,198.51.100.2:9092'` | | TOPIC | None | STRING | The name of the Kafka topic indicates where to consume records. | | WRITE\_OFFSETS | `true` | BOOLEAN | Optional.

Indicates whether the Kafka consumer should write its durably-made record offsets to the Kafka Broker. | | CONFIG | `'{`
`"group.id": "____"`
`}'` | JSON-formatted STRING | Optional.

The [consumer configuration](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html) that the Kafka consumers should use.

Frequently specified configurations include `group.id` and `security.protocol`.

These configurations are fixed and cannot be overridden:
`{`
`"enable.auto.commit": false,`
`"key.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer",`
`"value.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer"`
`}` | | AUTO\_OFFSET\_RESET | `'latest'` | STRING | Optional.

Determine which action to take for the Kafka configuration when there is no initial offset in the offset store or the specified offset is out of range.

Supported values are:

`'earliest'` — Automatically reset the offset value to the smallest value.

`'latest'` — Automatically reset the offset value to the largest value. | | GROUP\_ID | `'____'` | STRING | Optional.

Client group identifier for the Kafka source configuration. | For the consumer configuration, to create a secure connection from the Kafka consumer to a Kafka Broker, set the `"security.protocol"` key along with any SSL or SASL keys. If a certificate file is required, you must add it to the truststore used by the on all Loader Nodes. The truststore path must be identical on all Loader Nodes. The Kafka configuration can reference this truststore path. If you specify the `ssl.certificate.location` or `ssl.ca.location` consumer configuration, you must specify both of these configurations. Otherwise, the system throws an error. For example: `CONFIG '{"auto.offset.reset": "earliest","ssl.certificate.location":"/etc/blab/file1.txt","ssl.ca.location":"/etc/blab/file2.txt"}'` #### Continuous File Loading Source Options Specify these options for data pipelines that use `FILESYSTEM` or `S3` sources with the `CONTINUOUS` mode. For a Kafka source, do not use these options. **General File Monitor Options** | **Option Key** | **Default** | **Data Type** | **Description** | | ----------------- | ----------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | MONITOR | None | STRING | The type of monitor. Valid values are: `sqs` or `kafka`. | | POLLING\_INTERVAL | 10 SECONDS | TIME INTERVAL | Optional.

Unit of time to consume the event topic. Valid value range: 10 to 120 seconds. | | MAX\_FILES | 100 | INTEGER | Optional.

In the `BATCH` syntax, the maximum number of pending files to collect before starting a batch. Valid value range: 10 to 2000. | | TIMEOUT | 1 MINUTE | TIME INTERVAL | Optional.

In the `BATCH` syntax, the time to wait before starting a batch. Valid value range: 10 seconds to 30 minutes. | | LOOKBACK | 1 DAY | TIME INTERVAL | Optional.

In the `BATCH` syntax, the unit of time to look back for file deduplication. Valid value range: 0 to 48 hours. | **SQS Monitor Options** Use these options when you use `MONITOR sqs` for . | **Option Key** | **Default** | **Data Type** | **Description** | | ------------------- | ----------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | QUEUE\_URL | None | STRING | The URL of the target queue. For example: `http://localhost:32769/000000000000/queue1` | | REGION | `NULL` | STRING | Optional.

The region for the SQS operation for signing requests.

When unspecified, the region is inferred from `QUEUE_URL` by looking for `sqs.REGION` or `sqs-fips.REGION`. If the region cannot be inferred, it defaults to `us-east-1`. | | ENDPOINT | `NULL` | STRING | Optional.

The endpoint URL for the client. For example: `http://localhost:32769`

When unspecified, defaults to `https://sqs[-fips].REGION.amazonaws.com` if `REGION` was inferred or specified. Otherwise, this option is required. | | ACCESS\_KEY\_ID | `''` | STRING | Optional.

Access key identifier for SQS authentication.

If you specify this option, you must also specify the `SECRET_ACCESS_KEY` option. The Ocient System uses anonymous credentials when you specify an empty value for this option. | | SECRET\_ACCESS\_KEY | `''` | STRING | Optional.

The secret access key for SQS authentication. | **Kafka Monitor Options** Use these options when you use `MONITOR kafka` for Kafka. | **Option Key** | **Default** | **Data Type** | **Description** | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BOOTSTRAP\_SERVERS | None | STRING | The bootstrap servers for the Kafka consumer configuration. The string contains a list of brokers as a comma-separated list of the broker hostnames or broker hostnames and port number combinations, in the format `hostname:port`. | | TOPIC | None | STRING | The name of the Kafka topic indicates where to consume records. | | AUTO\_OFFSET\_RESET | `'latest'` | STRING | Optional.

Action to take for the Kafka configuration when there is no initial offset in the offset store or the chosen offset is out of range:
`'earliest'` — Automatically reset the offset to the smallest offset.
`'latest'` — Automatically reset the offset to the largest offset. | | GROUP\_ID | `'_____monitor'` | STRING | Optional.

Client group identifier for the Kafka configuration. | | CONFIG | `'{ "enable.auto.commit": false,`
`"group.id": "_____monitor",`
`"max.poll.interval.ms": "600000",`
`"heartbeat.interval.ms": "6000",`
`"session.timeout.ms": "600000" }'` | JSON-formatted STRING | Optional.

The [consumer configuration](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html) that the Kafka consumers should use. This option is a JSON-formatted string.
Certain values within this configuration are fixed, whereas the Ocient System provides other values with a default value that you can modify. | | MAX\_MESSAGES | 100 | INTEGER | Optional.

In the `CONSUMER` syntax, the maximum number of messages to poll each time. | | TIMEOUT | 1000 MILLISECONDS | TIME INTERVAL | Optional.

In the `CONSUMER` syntax, the operation timeout that controls how long the consume request waits for the response. The valid range is: 0 milliseconds to 1 hour. | | FILENAME | `$"Records"[1].s3.object.key`
corresponds to a `s3:ObjectCreated:Put` event. | SELECTOR | Optional.

In the `MESSAGE` syntax, defines the JSON selector for the filename of an incoming message in a monitor. The default value works for `s3:ObjectCreated:*` events. If you override this default configuration, you can parse custom JSON messages. The format must still adhere to S3 standards. For details, see [Amazon Simple Storage Service Event message structure](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html). The message must be in the JSON format. If you specify this option, then you must specify the other two options in the `MESSAGE` syntax. | | TIMESTAMP | `$"Records"[1]."eventTime"`
corresponds to a `s3:ObjectCreated:Put` event. | SELECTOR | Optional.

In the `MESSAGE` syntax, defines the JSON selector for the last modification timestamp of the file of an incoming message in a monitor. The default value works for `s3:ObjectCreated:*` events. If you override this default configuration, you can parse custom JSON messages. The format must still adhere to S3 standards. For details, see [Amazon Simple Storage Service Event message structure](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html). The message must be in the JSON format. If you specify this option, then you must specify the other two options in the `MESSAGE` syntax. | | SIZE | `$"Records"[1].s3.object.size`
corresponds to a `s3:ObjectCreated:Put` event. | SELECTOR | Optional.

In the `MESSAGE` syntax, defines the JSON selector for the file size of an incoming message in a monitor. The default value works for `s3:ObjectCreated:*` events. If you override this default configuration, you can parse custom JSON messages. The format must still adhere to S3 standards. For details, see [Amazon Simple Storage Service Event message structure](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html). The message must be in the JSON format. If you specify this option, then you must specify the other two options in the `MESSAGE` syntax. | The same Kafka `CONFIG` option override considerations apply. For details, see [KAFKA Source Options](#kafka-source-options). ### LOOKUP Options You can optionally look up data in a table from an external database. You can include this table in the `SELECT` statement of the `CREATE PIPELINE` SQL statement and perform join operations on its columns. To use the `LOOKUP` keyword, specify the external source name `lookup_source`. You can look up data in multiple external databases. In this case, use the `LOOKUP` keyword with the source name and corresponding options for each database. You must provide the appropriate JDBC JAR file for the JDBC connection to external databases. For tables in the Ocient System, this JAR file is not needed. The Ocient System executes the JDBC code as trusted code. Only provide trusted JAR files to the loading process. The `streamloader.extractorEngineParameters.configurationOption.engine.external.jdbc.jarRootDirectory` configuration parameter specifies the location of the JAR file. For details, see [Configuration Settings for Data Pipelines](/configuration-settings-for-data-pipelines). | **Option Key** | **Default** | **Data Type** | **Description** | | ------------------ | ----------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CONNECTION\_TYPE | None | STRING | The type of connection. Only `'jdbc'` is supported. | | CONNECTION\_STRING | None | STRING | The string for connection to the external database. For example: `'jdbc:sqlite:/path/to/file'` | | LOOKUP\_SCHEMA | None | STRING | The schema of the table in the external database. | | LOOKUP\_TABLE | None | STRING | The name of the table in the external database. | | REFRESH\_PERIOD | 30 | TIME INTERVAL | Optional.

The time between lookup cache refreshes. | | CONFIG | `NULL` | JSON-formatted STRING | Optional.

The JSON string that contains additional configuration options. For example: `'{"force_refresh_if_null": false}'`

Supported JSON keys are:

`force_refresh_if_null`, specified as a Boolean that defaults to `true`. When you set this value to `true`, the system triggers a lookup cache refresh for each value missing from the lookup table (to account for a concurrent load). This cache refresh can have a noticeable performance impact. As such, if you expect that some values cannot result in successful lookups, then set this key to the `false` value.

`load_timeout_seconds`, specified as a long integer with no default value. This option is the timeout, in seconds, for a lookup cache refresh.

`driver_class`, specified as a string with no default value. This option is the fully qualified JDBC driver class name. If you specify this option, the system throws an error when the system cannot find this driver or the driver cannot load into the process. If the JAR file for the connection is a JDBC 4 file, the system automatically loads the driver class. Otherwise, you must specify the fully qualified class name (from `Class::getName` in ) using this option, such as `'{"driver_class": "org.test.example.Driver"}'`. | ### EXTRACT Options #### General Extract Options You can specify these options on any of the allowed `FORMAT` types. | **Option Key** | **Default** | **Data Type** | **Description** | | ---------------------------- | ---------------------------------------------------------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | FORMAT | None | STRING | Specifies the format of the files to load. Supported values are:
`delimited`
`csv`
`json`
`binary`
`asn.1`
`parquet`
`xml` | | CHARSET\_NAME | Binary format: `ibm1047`
All other formats: `utf-8` | STRING | Optional.

Specifies the character set for decoding data from the source records into character data. Use this character set when you load data into VARCHAR columns or when you apply `CHAR` transformation functions.
Defaults to `utf-8` for all formats except BINARY, which defaults to `ibm1047`.
You can configure the default value for `BINARY` formatted data using a SQL statement such as:
`ALTER SYSTEM ALTER CONFIG SET 'sql.pipelineParameters.extract.binary.defaultCharset' = 'ibm1047'` | | COLUMN\_DEFAULT\_IF\_NULL | `false` | BOOLEAN | Optional.

Specifies whether pipelines should load the column default value when the result of a series of transforms is NULL.

If you set this option to `false` (the default), the pipeline loads NULL values into columns.

If you set this option to `true`, the pipeline loads the defined default value of the column when the result of the execution of the transformation on the column is NULL. | | NULL\_STRINGS | Delimited or CSV format:
`['null', 'NULL']`
JSON format:
`[]` | ARRAY OF STRINGS | Optional.

Specifies string values that should represent a NULL value when extracted from the source records.
Use this in `csv`, `delimited`, and `json` formats to convert specific values to NULL instead of requiring individual transform function calls to `NULL_IF` with those values.
This option applies to source data immediately after extraction. If the result of transformations is one of these strings, you must use `NULL_IF` to transform to NULL with the specified string values. | | TRIM\_WHITESPACE | `false` | BOOLEAN | Optional.

Specifies whether to trim whitespace from the beginning and end of each string field during extraction. In this case, whitespace is a space, tab, carriage return, or linefeed character.

If the trimmed field value matches one of the values specified by the `NULL_STRINGS` option, the system converts the value in the field to `NULL`. If you set this value to `true` and the `EMPTY_FIELD_AS_NULL` option to `true`, the system converts any field that contains only whitespace to `NULL`.

This option does not trim arrays of strings.

To trim elements in arrays, you can use the `TRIM` transform function or the `TRANSFORM` function with a Lambda expression that uses the `TRIM` function. | | VALIDATE\_CHARACTERS | `false` | BOOLEAN | Optional.

Specifies whether to detect invalid characters based on the encoding type during extraction. If the system detects invalid characters, the system throws a file-level error for file loads or a record-level error for Kafka loads.

This option is not supported for ASN.1, , and binary formats. | | REPLACE\_INVALID\_CHARACTERS | `false` | BOOLEAN | Optional.

Specifies whether to replace invalid characters based on the encoding type with the replacement character `U+FFFD`.

This option is not supported for ASN.1, Parquet, and binary formats. | | REPLACEMENT\_CHARACTER | `�` | STRING | Optional.

Specifies the character that indicates invalid data when you set the REPLACE\_INVALID\_CHARACTERS option to `true`. The default value is the replacement character `U+FFFD`.

This option is not supported for ASN.1, Parquet, and binary formats. | For ASN.1, , and Parquet extract options, the SCHEMA option is a `MAP` type with a `NULL` default value. This option is optionally specified depending on the format. SCHEMA specifies the syntax of options for schema retrieval. #### ASN.1 Extract Options You can specify these options for ASN.1 data record extraction (`FORMAT ASN.1`). For details about ASN.1-formatted data, see [Load ASN.1 Data](/data-formats-for-data-pipelines#load-asn-1-data). | **Option Key** | **Default** | **Data Type** | **Description** | | -------------- | ----------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | URL | None | STRING | In the `SCHEMA` syntax, specify the URL to the ASN.1 schema file in the `.asn` format. | | RECORD\_TYPE | None | STRING | In the `SCHEMA` syntax, specify the fully qualified name of the record type to parse. The name is the root-level ASN.1 type to extract from each DER-encoded record. (e.g., `Example.PersonnelRecord`) | #### Avro Extract Options You can specify these options for Avro data record extraction. The `SCHEMA` options are all optional for object container files (OCF). For Kafka loads, the schema configuration must include either the `URL` option or the `INLINE` option, but not both. For file-based loads, the schema configuration can include: * Either the `INLINE` option or the `INFER_FROM` option, but not both of these options * Neither the `INLINE` nor the `INFER_FROM` options For details about loading Avro-formatted data, see [Load Avro Data](/data-formats-for-data-pipelines#load-avro-data). | **Option Key** | **Default** | **Data Type** | **Description** | | ------------------------------ | -------------------------------------------------------------------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | URL | None | STRING | The URL of the schema registry. | | CONFIG | None | STRING | The configuration string that includes authentication for accessing the schema registry. | | SUBJECT | `-value`, where `` is the name of the source Kafka topic | STRING | The custom subject for the schema registry. | | SCHEMA\_REGISTRY\_ID\_LOCATION | `header_or_value` | STRING | For Kafka loads, the location to store the schema identifier for the schema registry. Supported values are:

`'header_or_value'` — Specifies that the system searches for the schema Universally Unique IDentifier (UUID) in the `__value_schema_id`
Kafka header. If the system finds the UUID, the system uses that identifier. Otherwise, the system derives the identifier from the starting bytes of the Kafka message value.

`'value'` — Specifies that the system derives a schema identifier or UUID from the starting bytes of the Kafka message value.

`'none'` — For inline schemas, specifies that there is no embedded schema identifier to skip. | | INLINE | None | STRING | The JSON specification of the schema declaration as a string. Specify either a JSON string or object. | | INFER\_FROM | None | STRING | For file-based loads, specifies that the system infers the schema from files. Supported value is: `'sample_file'` — Use a sample file. | #### BINARY Extract Options You can apply these options to binary data record extraction (`FORMAT BINARY`). For details about BINARY-formatted data, see [Load Binary Data](/data-formats-for-data-pipelines#load-binary-data). The general option `CHARSET_NAME` has a different default value for `FORMAT BINARY`. | **Option Key** | **Default** | **Data Type** | **Description** | | ------------------- | ----------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | RECORD\_LENGTH | None | INTEGER | Specifies the fixed size in bytes of each record in the source data. The Ocient System splits the binary data into binary chunks according to this length value and processes them individually. | | ENDIANNESS | `big` | STRING | Optional.

Specifies the endianness used to interpret multi-byte sequences in various transforms of binary data. Accepted values are `‘big'` and `'little'`. | | AUTO\_TRIM\_PADDING | `true` | BOOL | Optional.

Determines if padding characters should be trimmed after decoding the binary data into string data. If you set this option to `TRUE`, the Ocient System trims all instances of the `PADDING_CHARACTER` value from the end of a string after the system decodes the string from `BINARY` type. | | PADDING\_CHARACTER | (space) | STRING | Optional.

The padding character from the string after the system decodes the string from `BINARY` type.
You can change the default padding character for `BINARY` formatted data using a SQL statement such as: `ALTER SYSTEM ALTER CONFIG SET 'sql.pipelineParameters.extract.binaryFormat.defaultPaddingCharacter' = '*'` | The Ocient System trims the default padding character of a space from the end of the text data in binary data. #### **Delimited and CSV** Extract Options You can specify these options for delimited and CSV format data record extraction (`FORMAT DELIMITED` or `FORMAT CSV` data formats, which are aliases). For details about working with delimited and CSV data, see [Load Delimited and CSV Data](/data-formats-for-data-pipelines#load-delimited-and-csv-data). | **Option Key** | **Default** | **Data Type** | **Description** | | ------------------------------- | ---------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | FIELD\_DELIMITER | `,` | STRING
or
ARRAY OF STRINGS | Optional.

Specifies a character or list of possible characters for delimiting fields. The default value sets the field delimiter to only a comma. The value must be one byte.
The Ocient System automatically interprets the values you specify as C-style escaped strings. This means you do not need to specify an escape string (`e'some value'`) is not required to specify control characters. This differs from the default string behavior in Ocient. For details, see [String Literals and Escape Sequences](/data-query-language-dql-statement-reference#string-literals-and-escape-sequences).
`FIELD_DELIMITER` and `FIELD_DELIMITERS` are aliases.
**Examples:**
Use a tab character:
`FIELD_DELIMITER = '\t'`
Use a pipe character:
`FIELD_DELIMITER = '\|'`
Use either a pipe or a comma character:
`FIELD_DELIMITER = [',', '\|']` | | RECORD\_DELIMITER | `['\r\n', '\n']` | STRING
or
ARRAY OF STRINGS | Optional.

Specifies the string or an array of strings for delimiting records. The file is split into individual records using this character during processing. Common values include `'\r\n'` and `'\n'`. The value must be one or two bytes.
The Ocient System automatically interprets the values you specify as C-style escaped strings. An escape string (`e'some value'`) is not required to specify control characters. This differs from the default strings behavior in Ocient.
The system chooses the first specified delimiter and uses that delimiter for the rest of the data. Data with mixed delimiters is not supported.
For details, see [String Literals and Escape Sequences](/data-query-language-dql-statement-reference).
`RECORD_DELIMITER` and `RECORD_DELIMITERS` are aliases.
**Examples:**
Use a linefeed character:
`RECORD_DELIMITER = '\n'`
Use a carriage return and linefeed character sequence:
`RECORD_DELIMITER = '\r\n'` | | NUM\_HEADER\_LINES | `0` | INTEGER | Optional.

Specifies the number of header lines, typically 0 or 1. The Ocient System skips this number of lines and does not load them as data during file processing. Use this option when your data includes a row of header values. | | NUM\_FOOTER\_LINES | `0` | INTEGER | Optional.

Specifies the number of footer lines, typically 0 or 1. The Ocient System skips this number of lines starting from the end of the file and does not load them as data during file processing. Use this option when your data includes a row of footer values. | | OPEN\_ARRAY | `[` | STRING | Optional.

Specifies the character that indicates the start of an array in a `csv` or `delimited` field. Use this option to parse array data types.
Specify the `CLOSE_ARRAY` option also when using this option.
Set this option to NULL or `''` to turn off the detection of these control characters. If you set this option to either of these characters, the system also turns off the detection of these characters for the `CLOSE_ARRAY` option.
**Example:**
Convert source data such as `val1,"[1,2,3]",val2` to an array when referenced as `$2[]`.
`OPEN_ARRAY '['`
`CLOSE_ARRAY ']'`
`ARRAY_ELEMENT_DELIMITER ','` | | CLOSE\_ARRAY | `]` | STRING | Optional.

Specifies the character that indicates the end of an array in a `csv` or `delimited` field. Use this option to parse array data types.
Specify the `OPEN_ARRAY` option also when using this option.
Set this option to NULL or `''` to turn off the detection of these control characters. If you set this option to either of these characters, the system also turns off the detection of these characters for the `OPEN_ARRAY` option.
**Example:**
Convert source data such as `val1,"{1,2,3}",val2` to an array when referenced as `$2[]`.
`CLOSE_ARRAY '}'`
`OPEN_ARRAY '{'`
`ARRAY_ELEMENT_DELIMITER ','` | | ARRAY\_ELEMENT\_DELIMITER | `,` | STRING | Optional.

Specifies the character that separates values in an array within a `csv` or `delimited` field. Use this option to parse array data types.
Set this option to NULL or `''` to turn off the detection of these control characters.
**Example:**
Convert source data such as `val1,"[1;2;3]",val2` to an array when referenced as `$2[]`.
`ARRAY_ELEMENT_DELIMITER = ';'` | | OPEN\_OBJECT | `{` | STRING | Optional.

Specifies the character that indicates the start of a tuple in a field. Use this option to parse tuple data types.
Specify the `CLOSE_OBJECT` option also when using this option.
Set this option to NULL or `''` to turn off the detection of these control characters. If you set this option to either of these characters, the system also turns off the detection of these characters for the `CLOSE_OBJECT` option. | | CLOSE\_OBJECT | `}` | STRING | Optional.

Specifies the character that indicates the end of a tuple in a field. Use this option to parse tuple data types.
Specify the `OPEN_OBJECT` option also when using this option.
Set this option to NULL or `''` to turn off the detection of these control characters. If you set this option to either of these characters, the system also turns off the detection of these characters for the `OPEN_OBJECT` option. | | EMPTY\_FIELD\_AS\_NULL | `true` | BOOLEAN | Optional.

Specifies whether the Ocient System should extract an empty source field as NULL or a missing value. When this option is set to `true`, the Ocient System treats empty fields as NULL. Otherwise, the system treats fields as a missing value. For string-type fields, a missing value is equivalent to an empty string.
Define an empty field as two consecutive delimiters (e.g., the second field is empty in `abc,,xyz`). If a field is explicitly an empty string as indicated by quote characters, the Ocient System treats the field as an empty string, not an empty field (e.g., the second field is an empty string in `abc,"",xyz`).
The `CHAR($1)` transformation function handles both NULLs and empty strings and passes them through.
⚠️ Beware that the `NULL_IF($1, '')` transformation function directly loads a NULL for both NULLs \*and \*missing values. This transformation can override the behavior of `EMPTY_FIELD_AS_NULL`. | | FIELD\_OPTIONALLY\_ENCLOSED\_BY | `"` | STRING | Optional.

Also known as the “quote character,” this option specifies the character for optionally enclosing fields. Fields enclosed by this character can include delimiters, the enclosure character, or the escape character.
Set this option to NULL or `''` to turn off the detection of these control characters.
**Examples:**
Use a double quote character:
`FIELD_OPTIONALLY_ENCLOSED_BY = '"'`
Use a single quote character:
`FIELD_OPTIONALLY_ENCLOSED_BY = ''''` | | ESCAPE\_CHAR | `"` | STRING | Optional.

Specifies the escape character within fields enclosed by the `FIELD_OPTIONALLY_ENCLOSED_BY` option. Use this option to escape the enclosure character or escape character.
Set this option to NULL or `''` to turn off the detection of these control characters.
**Examples:**
Use a double quote as the escape character.
`ESCAPE_CHAR = '"'`
Use a single quote as the escape character. When you specify the escape character, you often have to use an escape sequence. This action follows standard SQL rules.
`ESCAPE_CHAR = ''''` | | SKIP\_EMPTY\_LINES | `false` | BOOLEAN | Optional.

Specifies whether or not to skip empty lines. | | COMMENT\_CHAR | `NULL` | STRING | Optional.

Specifies the character used to comment out a record in the source file. The load skips records where the first character of a record is equal to this character.
Set this option to NULL or `''` to turn off the detection of these control characters.
Example: `COMMENT_CHAR '#'` | | HEADERS | `NULL` | ARRAY OF STRINGS | Optional.

Specifies the header labels associated with each column in a delimited file. This array of values corresponds to the columns in order from left to right. Use these labels in the `CREATE PIPELINE SELECT` SQL statement to refer to column values. For example, if you specify `HEADERS ['col1', 'col2', col3']`, you can refer to the first column as `$"col1"` instead of `$1` in the `SELECT` statement. | | STRIP\_ARRAY\_ELEMENT\_QUOTES | `false` | BOOLEAN | Optional.

When you set this option to `true`, the system removes quote characters from array elements, so that, for example, `["str"]` becomes `[str]`. | | STRIP\_FIELD\_QUOTES | `true` | BOOLEAN | Optional.

When you set this option to `true`, the system removes quote characters from the string, so that, for example, `"str"` becomes `str`. | | TRIM\_ARRAY\_ELEMENTS | `false` | BOOLEAN | Optional.

When you set this option to `true`, the system removes whitespace characters from the array elements, so that, for example, `" str"` becomes `"str"`. | #### JSON Extract Options No options exist for JSON data record extraction (`FORMAT JSON`). For details about JSON-formatted data, see [Load JSON Data](/data-formats-for-data-pipelines#load-json-data). #### PARQUET Extract Options You can specify these options for Parquet data record extraction (`FORMAT ``PARQUET`). For details about Parquet-formatted data, see [Load Parquet Data](/data-formats-for-data-pipelines#load-parquet-data). When you use the `FORMAT PARQUET` option with an AWS S3 Source, the `ENDPOINT` option is required. | **Option Key** | **Default** | **Data Type** | **Description** | | ------------------ | ------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SCHEMA INFER\_FROM | `sample_file` | STRING | In the `SCHEMA` syntax, specify how to infer the Parquet schema. The supported value is:
`sample_file` — Infer from a random file.

**Example:**
`SCHEMA (INFER_FROM`
`sample_file)` | #### XML Extract Options No options exist for the XML format extraction (`FORMAT XML`). For details about XML-formatted data, see [Load XML Data](/data-formats-for-data-pipelines#load-xml-data). ### Bad Data Targets Bad data represents records that the Ocient System could not load due to errors in the transformations or invalid data in the source records. You can provide options for a bad data target that the Ocient System uses during pipeline execution to capture the records that are not loaded. The original bytes that the pipeline tried to load are captured in the bad data target along with the metadata about the error, such as the error message or source. Kafka is the only supported bad data target. #### Kafka Bad Data Target When you use Kafka as a bad data target, the Ocient System produces the original bytes of the source record into the Kafka topic of your choice. The Ocient System includes the metadata about the record in the header of the record as it is sent to Kafka. You can configure the Kafka topic on your Kafka Brokers using the retention and partition settings of your choice. In the event that the Kafka Broker is unreachable when the Ocient System attempts to produce a bad data record to the bad data target, the system logs an error on the Loader Node, and the pipeline continues. **Example** This example `CREATE PIPELINE` SQL statement snippet contains a bad data target definition using the `BAD_DATA_TARGET` option. ```sql SQL theme={null} CREATE PIPELINE ... BAD_DATA_TARGET KAFKA TOPIC 'orders_errors' BOOTSTRAP_SERVERS '111.11.111.1:9092,111.11.111.2:9092' CONFIG '{"compression.type": "gzip"}' SOURCE ... EXTRACT ... INTO public.orders SELECT $order.billing.name as username, $order.subtotal as subtotal, ... ``` #### Kafka Bad Data Target Options | **Option Key** | **Default** | **Data Type** | **Description** | | ------------------ | ---------------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BOOTSTRAP\_SERVERS | None | STRING | A comma-delimited list of `IP:port` pairs that contain the IP addresses of the Kafka Brokers and the associated port number.
**Example:**
`BOOTSTRAP_SERVERS = '111.11.111.1:9092,111.11.111.2:9092'` | | TOPIC | None | STRING | The name of the Kafka topic where the Ocient System should produce bad data records. | | CONFIG | `'{`
`"compression.type": "none"`
`}'` | JSON-formatted STRING | Optional.

The [producer configuration](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html) that the Kafka producer should use. | ### **Advanced Pipeline Tuning Options** You can use pipeline tuning options to control the parallelism or batching dynamics of your pipelines. This tuning can throttle the resources used on a pipeline or increase parallel processing across Loader Nodes. These options are advanced settings that might require a detailed understanding of the underlying mechanics of the loading infrastructure in the Ocient System to employ. Due to the inherent nature of each source type, the behavior of these options can differ between file-based and Kafka-based loads. All these options are optional. \| **Option Key** | **Default** | **Data Type** | **Description** | \| --- | --- | --- | --- | --- | \| CORES | The maximum number of CPU cores available on each Loader Node. | INTEGER | Maximum number of processing threads that the Ocient System uses during execution on each Loader Node. The system creates this number of threads on each Loader Node.
The Ocient System automatically determines the default value by finding the number of cores of a Loader Node. You can use this option for performance tuning.
The calculation for maximum parallelism of a pipeline is: `number_of_loaders * CORES`.
**About Kafka Partitions and Parallelism**
For Kafka Loads, this option determines the number of Kafka Consumers created on each Loader Node.
For Kafka Pipelines, the recommendation is that `number_of_loaders * CORES` equals the number of Kafka topic partitions.
If this number exceeds the number of Kafka topic partitions, the work might spread unevenly across Loader Nodes.
If this number is less than the number of Kafka topic partitions, some Kafka Consumers might receive uneven amounts of work. In this case, use a value for `number_of_loaders * CORES` that is an even divisor of the number of Kafka topic partitions to avoid a skew in the rates of processing across partitions. | \| PARTITIONS | Equal to the value of CORES. | INTEGER | Specifies the number of partitions over which to split the file list. Not applicable to Kafka loads.
The Ocient System automatically sets a default value based on the configured value for the `CORES` option. You can use this option for performance tuning.
The number of partitions determines how many buckets of work the Ocient System generates for each batch of files processed on a Loader Node. The pipeline processes this number of partitions in parallel using the specified number of cores.
If you specify fewer partitions than cores, some cores are not fully utilized, and resources are wasted. If you specify more partitions than cores, the Ocient System divides partitions in a round-robin fashion over the available cores. | \| BATCH\_SIZE | A dynamic value, determined by the Ocient System for each pipeline to maximize performance. | INTEGER | Number of rows in the batch to load at one time.
The Ocient System automatically calculates a dynamic value depending on the table columns and the utilization of internal buffers to transfer records to the database backend. You can use this option to turn off the dynamic adjustments for performance tuning.
⚠️ Only change this setting in rare cases where loading performance is slower than expected, and you have a large record size. If this setting is improperly set, pipelines might fail with out-of-memory exceptions.
You can configure the default value (for the batch payload target) using a SQL statement such as:
`ALTER SYSTEM ALTER CONFIG SET 'streamloader.extractorEngineParameters.configurationOption.osc.batch.payload.target' '65536'` | \| RECORD\_NUMBER\_FORMAT | For file loads that do not use the [EXPLODE\_OUTER](/special-data-pipeline-transformation-functions#explode_outer) function, the default is \`\[19, 45, 0]\`.
For Kafka loads that do not use the \`EXPLODE\_OUTER\` function, the default is \`\[0, 64, 0]\`.
For loads that use the \`EXPLODE\_OUTER\` function, the default is the specified load-type-specific default value with 13 subtracted from the record index bits. The system adds these bits to the bits for rows within a record.
For example, the default for file loads that use this function is \`\[19, 32, 13]\`. | ARRAY | The 64-bit record number for each record of the load. This number uniquely identifies a row within its partition.
The format is an array with three values in the format\`\[, , ]\`
The file index bits \`\` value is the number of bits used to represent the file index within a partition.
The record index bits \`\` value is the number of bits used to represent the record index within a file.
The rows per record index bits \`\` is the number of bits used to represent the row within a record. The system uses this value with the \`EXPLODE\_OUTER\` function.
These three values must sum to 64.
\*\*Example\*\*
\`RECORD\_NUMBER\_FORMAT= \[10, 54, 0]\`
Set the number of file index bits to \`10\` and the number of record index bits to \`54\`, allowing up to 2^10 files and 2^54 records per file. The system does not support the \`EXPLODE\_OUTER\` function in this configuration because the rows per record index bits are \`0\`. | | ## DROP PIPELINE `DROP PIPELINE` removes an existing pipeline in the current database. You cannot remove a pipeline that is running. You must have the DROP privilege on the pipeline to execute this SQL statement. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). When you drop a pipeline, the Ocient System also removes the associated system catalog information, such as pipeline errors, events, files, partitions, and metrics. **Syntax** ```sql SQL theme={null} DROP PIPELINE [ IF EXISTS ] pipeline_name [, ...] ``` | **Parameter** | **Data** **Type** | **Description** | | --------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `pipeline_name` | string | The name of the specified pipeline to remove.
You can drop multiple pipelines by specifying additional pipeline names, separated by commas. | **Examples** **Remove Existing Data Pipeline** Remove an existing pipeline named `ad_data_pipeline`. ```sql SQL theme={null} DROP PIPELINE ad_data_pipeline; ``` **Remove Existing Data Pipeline by Checking for Existence** Remove an existing pipeline named `ad_data_pipeline` or return a warning if the Ocient System does not find the pipeline in the database. ```sql SQL theme={null} DROP PIPELINE IF EXISTS ad_data_pipeline; ``` ## PREVIEW PIPELINE `PREVIEW PIPELINE` enables you to view the results of loading data for a specific `CREATE PIPELINE` SQL statement without creating a whole data pipeline and without storing those results in the target table. Using this SQL statement, you can iterate quickly and modify the syntax as needed to achieve your expected results. After you confirm your expected results, you can use the same syntax in the body of the `CREATE PIPELINE` statement with the appropriate source. A table must exist in the database to serve as the target of your `PREVIEW PIPELINE` statement. This table ensures the pipeline matches the column types of the target table. However, the execution of this statement does not load data into the target table. ### Preview Sources The `SOURCE INLINE` source type is available only for the `PREVIEW PIPELINE` SQL statement. You cannot create a data pipeline with inline source data. The inline source data limit is 1,000 rows. Other source types defined in the `CREATE PIPELINE` statement, `S3`, `KAFKA`, and `FILESYSTEM`, are compatible with the `PREVIEW PIPELINE` statement. The extract options vary by the source type to mirror the `CREATE PIPELINE` statement. The database returns 10 records by default. ### Preview Error Handling Pipeline-level errors cause the `PREVIEW PIPELINE` SQL statement to fail. The Ocient System returns an error and no result set. However, the Ocient System accumulates record-level errors that occur during the execution of this statement in a single warning that the system returns along with the result set. Each line of the warning describes a record-level error in human-readable form or as a JSON blob, depending on the value of the `SHOW_ERRORS_AS_JSON` option. Rows or columns that encounter record-level errors have `NULL` values in the result set. ### Preview Limitations Limitations of this SQL statement are: * Before executing a `PREVIEW PIPELINE` SQL statement, you must create a table for the Ocient System to have context for the preview. * The maximum number of rows a `PREVIEW PIPELINE` SQL statement can return is 1,000 rows. * The `COLUMN_DEFAULT_IF_NULL` option from the `CREATE PIPELINE` SQL statement has no effect on the `PREVIEW PIPELINE` SQL statement. * The `PREVIEW PIPELINE` SQL statement does not honor the assignment of a service class based on text matching. * These source options are not supported: * `START_FILENAME` * `END_FILENAME` * When you execute two duplicate `PREVIEW PIPELINE` statements for a specific Kafka topic, the two statements share a consumer group. If the topic is small, one or both of the result sets might only be a partial result. * For multiple tables, you can preview only one table at a time. You must specify the name of the table you want to preview using the `FOR` keyword. * Previewing a continuous data pipeline is not supported. **Syntax** ```sql SQL theme={null} PREVIEW PIPELINE pipeline_name [ MODE mode ] [ SHOW_ERRORS_AS_JSON show_errors_as_json ] SOURCE [ INLINE ] (inline_string | | | ) [ LIMIT limit ] EXTRACT FORMAT csv RECORD_DELIMITER record_delimiter FIELD_DELIMITERS ['delim1', 'delim2', ...] [ INTERMEDIATE_VALUES intermediate_values ] [ INSERT ] INTO created_tablename SELECT preview_column_formula AS preview_column_name, ... [ WHERE filter_expression ] [ [ INSERT ] INTO created_tablename_n SELECT preview_column_formula AS preview_column_name, ... [ WHERE filter_expression ] ] [ ,... ] [ FOR created_tablename_n ] ``` Though this syntax shows the CSV format, you can also use the `PREVIEW PIPELINE` statement with the other formats. | **Parameter** | **Data** **Type** | **Description** | | ------------------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pipeline_name` | string | The name of the specified data pipeline for the preview. | | `created_tablename` | identifier | The identifier for the name of the table that you create before executing the `PREVIEW PIPELINE` SQL statement. | | `created_tablename_n` | identifier | For multiple tables, the identifier for the name of another table that you create before executing the `PREVIEW PIPELINE` SQL statement. Use the `FOR` keyword to specify which table content to preview. | | `preview_column_formula` | identifier | The identifier for the formula of the data to load.
For example, for the data in the first field of the inline source, use `$1`.

If you need to add a transformation, you can use functions to transform data, such as `CONCAT($1, $2)`, to load the concatenation of the first two fields in the inline source data. | | `preview_column_name` | identifier | The name of the column in the target table. | **SQL Statement Options** | **Option Key** | **Default** | **Data** **Type** | **Description** | | ---------------------- | ------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SOURCE INLINE | Required | STRING | The string that contains data for the preview of the data pipeline load. For example, source data can be `'oci,ent,ocient\|ware,house,warehouse'`, where `\|` is the record delimiter and `,` is the field delimiter. For special characters, such as `\t`, use an escape sequence such as `e'oci,ent,oci\tent\|ware,house,ware\thouse'`.

This option supports these text-based formats: CSV, JSON, and XML. | | MODE | `'transform'` | STRING | Indicates whether to perform a validation of the PREVIEW PIPELINE SQL statement.

Valid values are: `'validate'` and `'transform'`.

Set this option to `'validate'` for checking that the creation of the data pipeline succeeds. If the pipeline is valid, the statement produces no output; otherwise, it returns an error.

Set this option to `'transform'` to retrieve a preview of the results of the pipeline. | | SHOW\_ERRORS\_AS\_JSON | `false` | BOOLEAN | Indicates whether to show errors. Values are `true` or `false`. If the value is `true`, the Ocient System returns record-level errors as JSON blobs rather than human-readable messages. | | LIMIT | 10 | INTEGER | The number of rows, specified as an integer, to return in the preview results for sources with many rows. The default value is 10 rows. | | INTERMEDIATE\_VALUES | `false` | BOOLEAN | Indicates whether to capture intermediate values during a transformation sequence. Values are `true` or `false`. If the value is `true`, the Ocient System appends an extra column to the result set. Each value in the column contains a JSON blob that describes the intermediate values processed for each column after each transformation. | You must specify at least one column name in the `SELECT` part of the syntax. The name of the specified column must match the name of the column in the created table. The number of columns in the `SELECT` part can be less than those in the created table. For definitions of other extract options, see the `CREATE PIPELINE` SQL statement options in [CREATE PIPELINE](#create-pipeline). **Examples** **Preview Pipeline Using CSV Format** Preview the load of two rows of data. First, create a table to serve as the context for the load. The `previewload` table contains three columns with these data types: string, integer, and Boolean. ```sql SQL theme={null} CREATE TABLE previewload (col1 VARCHAR, col2 INT, col3 BOOLEAN); ``` Create the preview pipeline `testpipeline` with this data: `'hello,2,true|bye,3,false'`. Specify the CSV extract format, `|` record delimiter, and the `,` field delimiter. Load the data without transformation. ```sql SQL theme={null} PREVIEW PIPELINE testpipeline SOURCE INLINE 'hello,2,true|bye,3,false' EXTRACT FORMAT CSV RECORD_DELIMITER '|' FIELD_DELIMITERS [','] INTO previewload SELECT $1 AS col1, $2 AS col2, $3 AS col3; ``` ## *Output* ```text Text theme={null} col1 col2 col3 \-------------------------------------------------------------- hello 2 true bye 3 false Fetched 2 rows ``` Delete the `previewload` table. ```sql SQL theme={null} DROP TABLE previewload; ``` **Preview Pipeline Using CSV Format with Escape Characters** Preview the load of two rows of data. First, create a table to serve as the context for the load. The `previewload` table contains three columns with these data types: string, integer, and Boolean. ```sql SQL theme={null} CREATE TABLE previewload (col1 VARCHAR, col2 INT, col3 BOOLEAN); ``` Create the preview pipeline `testpipeline` with this data: `'hello\tworld,2,true|bye\tworld,3,false'`. Specify the CSV extract format, `|` record delimiter, and `,` field delimiter. Load the data without transformation. In this case, the data contains the special character `\t`. You must escape the character by using the escape sequence `e`. ```sql SQL theme={null} PREVIEW PIPELINE testpipeline SOURCE INLINE e'hello\tworld,2,true|bye\tworld,3,false' EXTRACT FORMAT CSV RECORD_DELIMITER '|' FIELD_DELIMITERS [','] INTO previewload SELECT $1 AS col1, $2 AS col2, $3 AS col3; ``` ## *Output* ```text Text theme={null} col1 col2 col3 \-------------------------------------------------------------- hello world 2 true bye world 3 false Fetched 2 rows ``` Delete the `previewload` table. ```sql SQL theme={null} DROP TABLE previewload; ``` **Preview Pipeline Using CSV Format with Transformation** Create a table to serve as the context for the load. The `previewload` table contains three string columns. ```sql SQL theme={null} CREATE TABLE previewload (col1 VARCHAR, col2 VARCHAR, col3 VARCHAR); ``` Create the preview pipeline `testpipeline` with this data: `'hello,world|bye,world'`. Specify the CSV extract format, `|` record delimiter, and the `,` field delimiter. Load the data with a transformation to concatenate the two strings and return the result in the third column. ```sql SQL theme={null} PREVIEW PIPELINE testpipeline SOURCE INLINE 'hello,world|bye,world' EXTRACT FORMAT CSV RECORD_DELIMITER '|' FIELD_DELIMITERS [','] INTO previewload SELECT $1 AS col1, $2 AS col2, CONCAT($1,$2) AS col3; ``` ## *Output* ```text Text theme={null} col1 col2 col3 \--------------------------------------------------------------------------------------------------------------------------------------- hello world helloworld bye world byeworld Fetched 2 rows ``` The third column contains the concatenated result of the first two columns. Delete the `previewload` table. ```sql SQL theme={null} DROP TABLE previewload; ``` **Preview Pipeline Using the Kafka Source** Create the `previewload` table with these columns: * `id` — Non-NULL integer * `salut` — Non-NULL string * `name` — Non-NULL string * `surname` — Non-NULL string * `zipcode` — Non-NULL integer * `age` — Non-NULL integer * `rank` — Non-NULL integer ```sql SQL theme={null} CREATE TABLE previewload ( id INT NOT NULL, salut VARCHAR(3) NOT NULL, name VARCHAR(10) NOT NULL, surname VARCHAR(10) NOT NULL, zipcode INT NOT NULL, age INT NOT NULL, rank INT NOT NULL); ``` Create the preview pipeline `test_small_kafka_simple_csv`. Specify the `ddl_csv` topic. Indicate that the Kafka consumer should not write its durably-made record offsets to the Kafka Broker by using the `WRITE_OFFSETS` option set to `false`. Specify the bootstrap server as `servername:0000` and configuration options as `"auto.offset.reset": "earliest" `by using the `BOOTSTRAP_SERVERS` and `CONFIG` options, respectively. Limit the returned results to three rows by using the `LIMIT` option. Specify the CSV extract format and `\n` record delimiter by using the `FORMAT` and `RECORD_DELIMITER` extract options, respectively. ```sql SQL theme={null} PREVIEW PIPELINE test_small_kafka_simple_csv SOURCE KAFKA TOPIC 'ddl_csv' WRITE_OFFSETS false BOOTSTRAP_SERVERS 'servername:0000' CONFIG '{"auto.offset.reset": "earliest"}' LIMIT 3 EXTRACT FORMAT csv RECORD_DELIMITER '\n' INTO previewload SELECT INT($1) AS id, CHAR($2) AS salut, CHAR($3) AS name, CHAR($4) AS surname, INT($5) AS zipcode, INT($6) AS age, INT($7) AS rank; ``` *** ```text Text theme={null} id salut name surname zipcode age rank \----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 105 Mr Jmhsuxofspx Uaofgayjugb 85573 29 2 101 Mr Ijmmtbddkyh Yqbxqnkgidp 52393 43 1 109 Mr Bigohpwfwmr Qcxgakpkoeu 74420 1 3 Fetched 3 rows ``` ## START PIPELINE `START PIPELINE` begins the execution of the specified data pipeline that extracts data and loads it into the target tables specified by the `CREATE PIPELINE` SQL statement. When you execute the `START PIPELINE` SQL statement, the Ocient System creates a static list of files in the `sys.pipeline_files` system catalog table and marks them with the `PENDING` status. After the system assigns a file to an underlying task, the system marks the file as `QUEUED`. After the system verifies that the file exists, the system marks the file as `LOADING` to signify that a Loader Node has started reading the source data. Finally, upon successfully loading the file, the system transitions the status of the file to the terminal status `LOADED`. A Kafka pipeline never enters the `COMPLETED` state in the `information_schema.pipeline_status` view. Instead, the pipeline remains running after you start the pipeline until you stop it or the pipeline reaches the specified error limit using the `ERROR LIMIT` option. You must have the EXECUTE privilege on the pipeline and the INSERT privilege on any table that is a target in the pipeline. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). **Syntax** ```sql SQL theme={null} START PIPELINE pipeline_name [ ERROR [ LIMIT ] [ FILE_ERROR (FAIL | SKIP_MISSING_FILE | TOLERATE) ] ] [ USING LOADERS ] [ ON COMPLETION (NO_FLUSH | FLUSH_AND_WAIT | FLUSH_AND_RETURN) ] ``` | **Parameter** | **Data** **Type** | **Description** | | --------------- | ----------------- | ---------------------------------------- | | `pipeline_name` | string | The name of the specified data pipeline. | **SQL Statement Options** | **Option Key** | **Default** | **Data Type** | **Description** | | ---------------------------------- | ----------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ERROR LIMIT `` | 0 | INTEGER | Error log option that determines the number of record-level errors that can occur during the execution of a pipeline that the load tolerates before the whole pipeline execution fails.
`` is a number greater than or equal to -1. When you set `` to -1, the load tolerates an unlimited number of record-level errors.
By default, continuous pipelines tolerate an unlimited number of record-level errors, whereas batch pipelines tolerate zero errors.

With multiple target tables, record-level errors are specific to a table. For example, if a transformation for loading one table succeeds but another transformation for loading another table fails, the row corresponding to the successful transformation loads to the first table but not the other. | | ERROR FILE\_ERROR `` | `FAIL` | STRING | For pipelines that load data from S3 or local file sources, this error configuration option determines how to treat unrecoverable file-level errors. Examples of unrecoverable file-level errors are:

The file is listed when the pipeline starts, but is missing later during the load.

The Gzip file is corrupted and cannot be decompressed.

The file cannot be downloaded from the source.

Record-level error that is not tolerable occurs when tokenizing or transforming data in the file.

`` can be one of these keywords:
`FAIL` — Fail the whole pipeline because of a file-level error.
`SKIP_MISSING_FILE` — Only tolerate errors that occur due to missing files. If a file exists in the list when the pipeline starts but is missing later during the load, skip the file and continue with the next file.
`TOLERATE` — Tolerate all unrecoverable file-level errors. In this mode, the load also tolerates an unlimited number of record-level errors.
The `FAILED`, `SKIPPED`, and `LOADED_WITH_ERRORS` file statuses appear in the `sys.pipeline_files` system catalog tables, respectively, and indicate how the pipeline handled the file error. | | USING LOADERS `loader_names` | `NULL` | LIST OF STRINGS | Specify one or more names of Loader Nodes as a comma-separated list for executing the `START PIPELINE` SQL statement. If you do not use this option, the Ocient System uses all of the Loader Nodes that are active to execute the pipeline.
You can find node names in the `sys.nodes` system catalog table. | | ON COMPLETION `` | `NO_FLUSH` | STRING | Completion type option that specifies the behavior when the pipeline finishes loading. This option determines when the remaining pages are converted into Segments.
`NO_FLUSH` — Do not force a flush of pages. Rely on watermarks and timeouts to trigger final conversion to Segments.
`FLUSH_AND_WAIT` — Trigger a flush of pages, initiating final conversion to Segments. The pipeline blocks and waits for the conversion to Segments to complete before marking the pipeline as `COMPLETED`.
`FLUSH_AND_RETURN` — Trigger a flush of pages, initiating the final conversion to Segments. The Ocient System marks the pipeline as `COMPLETED` immediately following the flush without waiting for conversion to Segments to complete. | For the query to execute successfully, the specified node names must identify nodes that have: * `ACTIVE` operational status * `streamloader` role When you execute the `START PIPELINE` SQL statement, the Ocient System creates a static list of files only for batch pipelines and a dynamic list for continuous pipelines in the `sys.pipeline_files` system catalog table. **Examples** Start an existing pipeline named `ad_data_pipeline` with default settings. ```sql SQL theme={null} START PIPELINE ad_data_pipeline; ``` Start an existing pipeline named `ad_data_pipeline` with error tolerance (tolerate 10 errors before aborting the pipeline). For details about error tolerance, see [Error Tolerance in Data Pipelines](/error-tolerance-in-data-pipelines). ```sql SQL theme={null} START PIPELINE ad_data_pipeline ERROR LIMIT 10; ``` Data pipelines log a message for each pipeline error to the `sys.pipeline_errors` system catalog table, even if you do not specify the ERROR option. Use `BAD_DATA_TARGET` settings to capture the original source data. Start an existing pipeline named `ad_data_pipeline` using the Loader Node named `stream-loader1`. ```sql SQL theme={null} START PIPELINE ad_data_pipeline USING LOADERS "stream-loader1"; ``` To resume a pipeline with file loading, see [Data Pipeline Behavior Considerations](/data-pipeline-behavior-considerations). To restart a Kafka data pipeline, see [Data Pipeline Behavior Considerations](/data-pipeline-behavior-considerations). For pipeline dependencies, see [Data Pipeline Behavior Considerations](/data-pipeline-behavior-considerations). ## STOP PIPELINE `STOP PIPELINE` stops the execution of the pipeline and its associated tasks. After you stop a pipeline, you can execute the `START PIPELINE` SQL statement on the pipeline to run the pipeline again. Regardless, the load deduplicates any records previously loaded in the same pipeline. You must have the EXECUTE privilege on the pipeline to execute this SQL statement. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). **Syntax** ```sql SQL theme={null} STOP PIPELINE pipeline_name ``` | **Parameter** | **Data** **Type** | **Description** | | --------------- | ----------------- | ---------------------------------------- | | `pipeline_name` | string | The name of the specified data pipeline. | **Example** Stop an existing pipeline named `ad_data_pipeline`. ```sql SQL theme={null} STOP PIPELINE ad_data_pipeline; ``` You can see the status of the parent tasks in the `sys.tasks` system catalog table and see the status of the child tasks in the `sys.subtasks` system catalog table. ## ALTER PIPELINE `ALTER PIPELINE` evolves the schema of the data pipeline in the `SELECT` clause. For pipelines with multiple tables, specify each `SELECT` clause for tables in the existing pipeline. The `WHERE` clause must match the corresponding clause of the existing pipeline. Use the `FORCE` keyword to change a non-backward-compatible schema or add, modify, or remove the `WHERE` clause. For details about schema evolution, see [Load Avro Data](/data-formats-for-data-pipelines). To execute this SQL statement, the data pipeline must not be running. You can use the `STOP PIPELINE` SQL statement to stop the execution of the data pipeline. You must have the ALTER privilege on the pipeline to execute this SQL statement. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). **Syntax** ```sql SQL theme={null} ALTER PIPELINE [ IF EXISTS ] pipeline_name [ INSERT ] INTO table_name SELECT $field_name AS field_name [ , ... ] [ WHERE ... ] [ INSERT INTO table_nameN SELECT ... ] [ FORCE ] ``` | **Parameter** | **Data** **Type** | **Description** | | --------------- | ----------------- | -------------------------------------------- | | `pipeline_name` | string | The name of the specified pipeline to alter. | **Examples** These examples use this table and data pipeline definition. You must stop the execution of the data pipeline before performing schema evolution operations. Create the `users` table with these columns: * `id` — Universally Unique IDentifier (UUID) of the user * `firstname` — First name of the user * `lastname` — Last name of the user * `birthyear` — Year of birth * `groups` — List of groups where the user belongs ```sql SQL theme={null} CREATE TABLE users( id UUID NOT NULL, firstname VARCHAR(255) NOT NULL, lastname VARCHAR(255) NOT NULL, birthyear INT, groups VARCHAR(255)[] NOT NULL DEFAULT 'char[]' ); ``` Assume you have user data in Avro format in multiple files in the `/data/users` directory. Create the `users_pipeline` data pipeline for the Avro files containing user data `*.avro`. The schema configuration instructs the system to infer from all files using the `INFER_FROM` option. ```sql SQL theme={null} CREATE PIPELINE users_pipeline SOURCE filesystem FILTER '/data/users/*.avro' EXTRACT FORMAT avro SCHEMA { INFER_FROM all_files } INTO users SELECT $id AS id, $firstname AS firstname, $lastname AS lastname, $birthyear AS birthyear; ``` Start the data pipeline. ```sql SQL theme={null} START PIPELINE users_pipeline; ``` **Schema Evolution with a Column Addition** Stop the data pipeline. ```sql SQL theme={null} STOP PIPELINE users_pipeline; ``` Change the schema of an existing pipeline named `users_pipeline` to add the `groups` column. Access the array of strings within the column. ```sql SQL theme={null} ALTER PIPELINE users_pipeline INTO users SELECT $id AS id, $firstname AS firstname, $lastname AS lastname, $birthyear AS birthyear, $groups[] AS groups; ``` Start the data pipeline again. ```sql SQL theme={null} START PIPELINE users_pipeline; ``` **Check Whether the Data Pipeline to Change Exists for Column Addition** Stop the data pipeline. ```sql SQL theme={null} STOP PIPELINE users_pipeline; ``` Change the schema of an existing pipeline named `users_pipeline` to add the `groups` column. Use the `IF EXISTS` keywords to check whether the data pipeline exists. Access the array of strings within the column. ```sql SQL theme={null} ALTER PIPELINE IF EXISTS users_pipeline INTO users SELECT $id AS id, $firstname AS firstname, $lastname AS lastname, $birthyear AS birthyear, $groups[] AS groups; ``` Start the data pipeline again. ```sql SQL theme={null} START PIPELINE users_pipeline; ``` **Schema Evolution with a Column Removal** Stop the data pipeline. ```sql SQL theme={null} STOP PIPELINE users_pipeline; ``` Change the schema of an existing pipeline named `users_pipeline` to remove the `birthyear` column. ```sql SQL theme={null} ALTER PIPELINE users_pipeline INTO users SELECT $id AS id, $firstname AS firstname, $lastname AS lastname; ``` Start the data pipeline again. ```sql SQL theme={null} START PIPELINE users_pipeline; ``` **Schema Evolution with a Column Data Type Modification** Stop the data pipeline. ```sql SQL theme={null} STOP PIPELINE users_pipeline; ``` Change the schema of an existing pipeline named `users_pipeline` to narrow the `INT` data type to a `SMALLINT` type for the `birthyear` column using the `SMALLINT` casting function. ```sql SQL theme={null} ALTER PIPELINE users_pipeline INTO users SELECT $id AS id, $firstname AS firstname, $lastname AS lastname, SMALLINT($birthyear) AS birthyear; ``` Start the data pipeline again. ```sql SQL theme={null} START PIPELINE users_pipeline; ``` **Schema** **Evolution with the Addition of a Filter Condition** Stop the data pipeline. ```sql SQL theme={null} STOP PIPELINE users_pipeline; ``` Change the schema of an existing pipeline named `users_pipeline` to filter the year of birth to be greater than 1950 using the `WHERE` clause. Add the `FORCE` keyword to instruct the data pipeline to evolve the schema with the filter condition. ```sql SQL theme={null} ALTER PIPELINE users_pipeline INTO users SELECT $id AS id, $firstname AS firstname, $lastname AS lastname, $birthyear AS birthyear, WHERE $birthyear > 1950 FORCE; ``` Start the data pipeline again. ```sql SQL theme={null} START PIPELINE users_pipeline; ``` ### ALTER PIPELINE RENAME `ALTER PIPELINE RENAME TO` SQL statement changes the name of the pipeline object, while retaining its identifier, options, and other metadata. The Ocient System reflects this change in the `sys.pipelines` system catalog table. Then, you must use the new name when you refer to the pipeline in SQL statements. You must have the ALTER privilege on the pipeline to execute this SQL statement. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). **Syntax** ```sql SQL theme={null} ALTER PIPELINE [ IF EXISTS ] pipeline_original_name RENAME TO pipeline_new_name ``` | **Parameter** | **Data** **Type** | **Description** | | ------------------------ | ----------------- | --------------------------------------- | | `pipeline_original_name` | string | The name of the existing data pipeline. | | `pipeline_new_name` | string | The new name of the data pipeline. | **Example** Rename an existing pipeline named `ad_data_pipeline` to `renamed_pipeline`. ```sql SQL theme={null} ALTER PIPELINE ad_data_pipeline RENAME TO renamed_pipeline; ``` ## **EXPORT** PIPELINE `EXPORT PIPELINE` returns the `CREATE PIPELINE` SQL statement used to create the pipeline object. You can use the output of this statement to recreate an identical pipeline when you remove the original pipeline. The execution of this statement censors sensitive S3 values like `ACCESS_KEY_ID` and `SECRET_ACCESS_KEY` and Kafka [Consumer Configuration](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html) password-type fields. The database replaces them with `*`. To execute this statement, you must have the VIEW privilege on the pipeline and any table the pipeline targets. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). **Syntax** ```sql SQL theme={null} EXPORT PIPELINE pipeline_name ``` | **Parameter** | **Data** **Type** | **Description** | | --------------- | ----------------- | ---------------------------------------- | | `pipeline_name` | string | The name of the specified data pipeline. | **Example** Export an existing pipeline in the database `ad_data_pipeline`. ```sql SQL theme={null} EXPORT PIPELINE ad_data_pipeline; ``` ## CREATE PIPELINE FUNCTION `CREATE PIPELINE FUNCTION` enables you to define a function for loading data. Define the function behavior using the language. For details bout this language, see [The Apache Groovy Programming Language](https://groovy-lang.org/index.html). Function arguments and output are strongly typed and immutable. You can test the execution of your function using the `PREVIEW PIPELINE` SQL statement. The Ocient System does not support the overload of function names. **Syntax** ```sql SQL theme={null} CREATE [ OR REPLACE ] PIPELINE FUNCTION [ IF NOT EXISTS ] function_name( input_argument [, ...] ) LANGUAGE GROOVY RETURNS output_argument_definition IMPORTS [ library_name [, ...] ] AS $$ groovy_declaration $$ ``` | **Parameter** | **Type** | **Description** | | ---------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `function_name` | string | A unique identifier for the data pipeline function. | | `input_argument` | string | The name of one or more input arguments of the function. Specify data types for input arguments according to the support data types defined in [Data Types for Data Pipelines](/data-types-for-data-pipelines).
For the data type declaration, use `NOT NULL` where applicable for maximum performance. | | `output_argument_definition` | string | The type definition of the output from the function. | | `library_name` | string | The name of one or more Java libraries.
You can include libraries by using the `IMPORTS` clause or specifying the fully-qualified class (e.g., `java.lang.Integer`) path in the source definition. | | `groovy_declaration` | string | The Groovy definition of the function. | ### Install and Enable Third-Party Libraries You can use the default list of supported third-party libraries or additional third-party libraries that you install. #### Supported Libraries Data pipeline functions can import classes from the default list of supported third-party libraries. This table provides the resources for each supported library package. | **Library Package** | **Resource** | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `java.lang.*` | [java.lang](https://docs.oracle.com/javase/8/docs/api/java/lang/package-summary.html) | | `java.util.*` | [java.util](https://docs.oracle.com/javase/8/docs/api/java/util/package-summary.html) | | `java.nio.ByteBuffer.*` | [ByteBuffer](https://docs.oracle.com/javase/8/docs/api/java/nio/ByteBuffer.html) | | `groovy.json.*` | [groovy.json](https://docs.groovy-lang.org/latest/html/gapi/groovy/json/package-summary.html) | | `groovy.xml.*` | [groovy.xml](https://docs.groovy-lang.org/latest/html/gapi/groovy/xml/package-summary.html) | | `groovy.yaml.*` | [groovy.yaml](https://docs.groovy-lang.org/latest/html/api/groovy/yaml/package-summary.html) | | `org.apache.groovy.datetime.extensions.*` | [org.apache.groovy.datetime.extensions](https://docs.groovy-lang.org/latest/html/api/org/apache/groovy/datetime/extensions/package-summary.html) | | `org.apache.groovy.dateutil` | [org.apache.groovy.dateutil.extensions](https://docs.groovy-lang.org/latest/html/api/org/apache/groovy/dateutil/extensions/package-summary.html) | | `com.ocient.streaming.data.types.*` | [Data Types for User-Defined Data Pipeline Functions](/data-types-for-user-defined-data-pipeline-functions) | #### Additional Libraries You can install and enable additional third-party libraries to import for use in your data pipeline functions. You must install the JAR package on all Loader Nodes in the `/opt/ocient/current/lib/extractorengine_udt` folder. Then, add the fully qualified class name in the function import list as part of the `library_name` parameter. For example, to reference the `ByteBuffer` class from the `com.fastbuffer` package, specify `com.fastbuffer.ByteBuffer` in the `library_name` parameter and use the class in the Groovy definition as `var x = new com.fastbuffer.ByteBuffer()`. ### Groovy Data Type Mapping For the Groovy definition, the Ocient System maps its SQL data type to the corresponding Groovy data type. Your Groovy code should use the Groovy data type defined in this table for any input arguments and output. | **SQL Data Type** | **Groovy Data Type** | | -------------------------- | -------------------------------------------------- | | `BIGINT` | `java.lang.Long` | | `BINARY(N) or HASH(N)` | `byte[]` | | `BOOLEAN` | `java.lang.Boolean` | | `CHAR(N) or VARCHAR(N)` | `java.lang.String` | | `DATE` | `java.time.LocalDate` | | `DECIMAL(P,S)` | `com.ocient.streaming.data.types.Decimal` | | `DOUBLE` | `java.lang.Double` | | `INT` | `java.lang.Integer` | | `IPV4` | `java.net.Inet4Address` | | `IP` | `java.net.Inet6Address` | | `ST_POINT` | `com.ocient.streaming.data.types.gis.STPoint` | | `ST_LINESTRING` | `com.ocient.streaming.data.types.gis.STLinestring` | | `ST_POLYGON` | `com.ocient.streaming.data.types.gis.STPolygon` | | `FLOAT` | `java.lang.Float` | | `SMALLINT` | `java.lang.Short` | | `TIME` | `com.ocient.streaming.data.types.Time` | | `TIMESTAMP` | `com.ocient.streaming.data.types.Timestamp` | | `BYTE` | `java.lang.Byte` | | `TUPLE<>` | `com.ocient.streaming.data.types.OcientTuple` | | `TYPE[]` | `java.util.List` | | `UUID` | `java.util.UUID` | | `VARBINARY(N)` | `byte[]` | | `VARCHAR(N)` | `java.lang.String` | **Example** Create the `sort_function` data pipeline function to sort an array of integers. The function has two input arguments: `value`, a non-NULL array of integers, and `ascending`, the sort order. The function returns a non-NULL array of integers. If `value` is empty, the function throws an error. The function imports these Java libraries: * `java.lang.Integer` * `java.util.ArrayList` * `java.util.Collections` * `java.util.Comparator` * `java.util.List` Define the Groovy code. Because the input arguments do not change, the example Groovy code first copies the `value` argument, sorts the copied list according to the sort order, and returns the sorted array. ```sql SQL theme={null} CREATE PIPELINE FUNCTION sort_function( value INT[] NOT NULL, ascending BOOLEAN NOT NULL) LANGUAGE GROOVY RETURNS INT[] NOT NULL IMPORTS [ 'java.lang.Integer', 'java.util.ArrayList', 'java.util.Collections', 'java.util.Comparator', 'java.util.List' ] AS $$ /* Throw an error if the array is empty */ if (value.isEmpty()){ throw new PipelineFunctionException("Unexpected empty array"); } /* Make a copy of the list. */ List sorted = new ArrayList<>((List)value); /* Sort the array elements according to the specified order. */ sorted.sort(ascending ? Comparator.naturalOrder() : Comparator.reverseOrder()); /* Return the sorted array. */ return sorted; $$; ``` View the creation information about the `sort_function` function using the `sys.pipeline_functions` system catalog table. This statement returns the function name, return type, argument names, data types of the arguments, and the list of imported libraries. ```sql SQL theme={null} SELECT name, return_type, argument_names, argument_types, imported_libraries FROM sys.pipeline_functions; ``` ## DROP PIPELINE FUNCTION `DROP PIPELINE FUNCTION` removes an existing pipeline function. You must have the DROP privilege on the pipeline function to execute this SQL statement. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). **Syntax** ```sql SQL theme={null} DROP PIPELINE FUNCTION [ IF EXISTS ] function_name [, ...] ``` | **Parameter** | **Data** **Type** | **Description** | | --------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `function_name` | string | The name of the specified data pipeline function to remove.
You can drop multiple pipelines by specifying additional function names, separated by commas. | **Examples** **Remove the Existing Pipeline Function** Remove an existing pipeline function named `sort_function`. ```sql SQL theme={null} DROP PIPELINE FUNCTION sort_function; ``` **Remove an Existing Pipeline Function by Checking for Existence** Remove an existing pipeline function named `sort_function` or return a warning if the Ocient System does not find the function in the database. ```sql SQL theme={null} DROP PIPELINE FUNCTION IF EXISTS sort_function; ``` ## Related Links [Load Data](/load-data) [Data Pipeline Load of CSV Data from S3](/data-pipeline-load-of-csv-data-from-s3) [Data Pipeline Load of Parquet Data from S3](/data-pipeline-load-of-parquet-data-from-s3) [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) [Data Pipeline Behavior Considerations](/data-pipeline-behavior-considerations) [Load Metadata and File-Based Partitioned Data in Data Pipelines](/load-metadata-and-file-based-partitioned-data-in-data-pipelines) [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference) [Identifiers](/identifiers) # Data Preparation Source: https://docs.ocient.com/data-preparation Use Ocient SQL lag generators and vector assemblers to simplify time series feature engineering for regression, VAR, and neural network model inputs. Lag generator and vector assembler SQL functions simplify generating and organizing lagged features for time series data. Each function expands into standard SQL during query processing, so developers can reduce boilerplate and create feature sets with minimal code. These functions fall into two main types: * [Lag Generator Functions](#lag-generator-functions) — Create individual lag feature columns. * [Vector Assembler Functions](#vector-assembler-function) — Group generated lag columns into vectors. Developers can apply lag features across many -supported models, including: * Simple Linear Regression (univariate forecasting with lag inputs) * Multiple Linear Regression (multivariate regression with lagged predictors) * Polynomial Regression (regression with higher-order lagged terms) * Linear Combination Regression (custom functions of lagged inputs) * Nonlinear Regression (arbitrary functions that include lag variables) * Feedforward Neural Networks (lagged features as input vectors) * Vector Autoregression (VAR) (multivariate time series with structured lag vectors) * Autoregression (univariate time series modeled through lagged values) * Vector Valued Regression (multivariate dependent variables with lagged inputs) The functions provide general-purpose utilities for time series feature engineering. In particular, VAR models benefit the most from lag functions because the models require large numbers of structured lag vectors. ## Lag Generator Functions Lag generator functions provide options for creating lagged columns for one or more variables in a single statement. These functions follow the syntax rules of window aggregate functions as they define the window for the lag computation with an `OVER` and `ORDER BY` clause. For details, see [Window Aggregate Functions](/window-aggregate-functions). The window parameters of lag generator functions are represented in these examples by ``. ### LAGS Generates a series of lagged columns for a single variable in one statement. This function simplifies univariate time-series feature creation. **Syntax** ```sql SQL theme={null} LAGS(expression, start_lag, end_lag [, step]) OVER () ``` | **Argument** | **Type** | **Description** | | ------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `expression` | ANY | The column or expression to lag. | | `start_lag` | INTEGER | First lag to generate. This value must be less than the `end_lag` value. | | `end_lag` | INTEGER | Last lag to generate. This value must be greater than the `start_lag` value. | | `step` | INTEGER | Optional.
Defines the gap between successive lags. Must be a positive integer.
If unspecified, this value defaults to `1`. | **Example** This query produces three lagged columns (`sales_lag1`, `sales_lag2`, `sales_lag3`) from the sales column. ```sql SQL theme={null} SELECT LAGS(sales, 1, 3) OVER (ORDER BY day) FROM store_sales; ``` *Output* ```sql SQL theme={null} sales_lag1 | sales_lag2 | sales_lag3 -----------+------------+------------ NULL | NULL | NULL 100 | NULL | NULL 120 | 100 | NULL 130 | 120 | 100 ``` ### LAGS\_ZEROFILL Generates lagged columns for a single variable and replaces NULL values with `0`. **Syntax** ```sql SQL theme={null} LAGS_ZEROFILL(expression, start_lag, end_lag [, step]) OVER () ``` | **Argument** | **Type** | **Description** | | ------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `expression` | ANY | The column or expression to lag. | | `start_lag` | INTEGER | First lag to generate. This value must be less than the `end_lag` value. | | `end_lag` | INTEGER | Last lag to generate. This value must be greater than the `start_lag` value. | | `step` | INTEGER | Optional.
Defines the gap between successive lags. Must be a positive integer.
If unspecified, this value defaults to `1`. | **Example** This query produces three lagged sales columns and replaces NULL values with 0. ```sql SQL theme={null} SELECT LAGS_ZEROFILL(sales, 1, 3) OVER (ORDER BY day) FROM store_sales; ``` *Output* ```sql SQL theme={null} sales_lag1 | sales_lag2 | sales_lag3 -----------+------------+------------ 0 | 0 | 0 100 | 0 | 0 120 | 100 | 0 130 | 120 | 100 ``` ### MULTI\_LAGS Generates lagged columns for multiple variables at once. This function is intended primarily for creating features for multivariate time-series models. **Syntax** ```sql SQL theme={null} MULTI_LAGS(expr1, expr2, [ ,... ] start_lag, end_lag [, step]) OVER () ``` | **Argument** | **Type** | **Description** | | ------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `expr1, expr2, [ ,... ] ` | ANY | One or more columns or expressions to lag. Applies the same lag range and step to each input. | | `start_lag` | INTEGER | First lag to generate. This value must be less than the `end_lag` value. | | `end_lag` | INTEGER | Last lag to generate. This value must be greater than the `start_lag` value. | | `step` | INTEGER | Optional.
Defines the gap between successive lags. Must be a positive integer.
If unspecified, this value defaults to `1`. | **Example** This query creates four lagged columns for both `interest_rate` and `gdp_growth`. ```sql SQL theme={null} SELECT MULTI_LAGS(interest_rate, gdp_growth, 1, 4) OVER (ORDER BY quarter) FROM economic_data; ``` *Output* ```sql SQL theme={null} interest_rate_lag1 | interest_rate_lag2 | interest_rate_lag3 | interest_rate_lag4 | gdp_growth_lag1 | gdp_growth_lag2 | gdp_growth_lag3 | gdp_growth_lag4 -------------------+--------------------+--------------------+--------------------+-----------------+-----------------+-----------------+----------------- NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL 2.5 | NULL | NULL | NULL | 1.1 | NULL | NULL | NULL 2.7 | 2.5 | NULL | NULL | 1.2 | 1.1 | NULL | NULL 2.8 | 2.7 | 2.5 | NULL | 1.3 | 1.2 | 1.1 | NULL ``` ### MULTI\_LAGS\_ZEROFILL Generates lagged columns for multiple variables and replaces NULL values with `0`. **Syntax** ```sql SQL theme={null} MULTI_LAGS_ZEROFILL(expr1, expr2, [ ,... ] start_lag, end_lag [, step]) OVER () ``` | **Argument** | **Type** | **Description** | | ------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `expr1, expr2, [ ,... ] ` | ANY | One or more columns or expressions to lag. Applies the same lag range and step to each input. | | `start_lag` | INTEGER | First lag to generate. This value must be less than the `end_lag` value. | | `end_lag` | INTEGER | Last lag to generate. This value must be greater than the `start_lag` value. | | `step` | INTEGER | Optional.
Defines the gap between successive lags. Must be a positive integer.
If unspecified, this value defaults to `1`. | **Example** This query generates four lagged columns each for `x1`, `x2`, and `x3`, replacing missing values with 0. ```sql SQL theme={null} SELECT MULTI_LAGS_ZEROFILL(x1, x2, x3, 1, 4) OVER (ORDER BY time_col) FROM public.my_time_series_table; ``` *Output* ```sql SQL theme={null} x1_lag1 | x1_lag2 | x1_lag3 | x1_lag4 | x2_lag1 | x2_lag2 | x2_lag3 | x2_lag4 | x3_lag1 | x3_lag2 | x3_lag3 | x3_lag4 --------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+--------- 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 10 | 0 | 0 | 0 | 20 | 0 | 0 | 0 | 30 | 0 | 0 | 0 11 | 10 | 0 | 0 | 21 | 20 | 0 | 0 | 31 | 30 | 0 | 0 12 | 11 | 10 | 0 | 22 | 21 | 20 | 0 | 32 | 31 | 30 | 0 ``` ## Vector Assembler Function A vector assembler function provides options for grouping multiple lagged columns into structured vectors in a single statement. This function makes preparing input data for time series models easier, especially those that require organized lag structures, such as VAR. ### LAG\_VECTORS Groups lagged columns generated by the `MULTI_LAGS` or `MULTI_LAGS_ZEROFILL` functions into vector columns. Models such as VAR require these structured lag vectors. The resulting vectors are named `lag_vector_`. **Syntax** ```sql SQL theme={null} LAG_VECTORS(expr1, expr2, [ ..., ] start_lag, end_lag [, step]) ``` | **Argument** | **Type** | **Description** | | ------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `expr1, expr2, [ ,... ] ` | NUMERIC | Two or more lagged columns to group. Applies the same lag range and step to each input. | | `start_lag` | INTEGER | First lag to assemble. This value must be less than the `end_lag` value. | | `end_lag` | INTEGER | Last lag to assemble. This value must be greater than the `start_lag` value. | | `step` | INTEGER | Optional.
Defines the gap between successive lags. Must be a positive integer.
If unspecified, this value defaults to `1`. | **Examples** **Assemble Lagged Columns into Vectors** This query uses `MULTI_LAGS_ZEROFILL` to generate lagged columns for `x1`, `x2`, and `x3`, then uses `LAG_VECTORS` to group them into `lag_vector1` through `lag_vector4`. ```sql SQL theme={null} SELECT {x1, x2, x3} AS current_vector, LAG_VECTORS(x1, x2, x3, 1, 4) FROM ( SELECT x1, x2, x3, MULTI_LAGS_ZEROFILL(x1, x2, x3, 1, 4) OVER (ORDER BY time_col) FROM public.my_time_series_table ) s; e; ``` *Output* ```sql SQL theme={null} current_vector | lag_vector1 | lag_vector2 | lag_vector3 | lag_vector4 ---------------+-------------+-------------+-------------+------------- {10, 20, 30} | {0, 0, 0} | {0, 0, 0} | {0, 0, 0} | {0, 0, 0} {11, 21, 31} | {10, 20, 30}| {0, 0, 0} | {0, 0, 0} | {0, 0, 0} {12, 22, 32} | {11, 21, 31}| {10, 20, 30}| {0, 0, 0} | {0, 0, 0} {13, 23, 33} | {12, 22, 32}| {11, 21, 31}| {10, 20, 30}| {0, 0, 0} ``` **Prepare Data for a VAR Model** This example shows how to prepare training data for a VAR model with three variables (`x1`, `x2`, `x3`) and four lags. The example demonstrates how the lag generator functions and vector assembler can simplify feature creation compared to writing out every LAG expression manually. * `MULTI_LAGS_ZEROFILL` generates all individual lagged columns (`x1_lag1`…`x3_lag4`) in a single call, replacing NULL values with zero. * `LAG_VECTORS` automatically groups these lagged columns into vector inputs (`lag_vector1` ... `lag_vector4`), which the model requires. ```sql SQL theme={null} CREATE MLMODEL my_var_model TYPE VECTOR_AUTOREGRESSION ON ( SELECT {x1, x2, x3}, LAG_VECTORS(x1, x2, x3, 1, 4) FROM ( SELECT x1, x2, x3, MULTI_LAGS_ZEROFILL(x1, x2, x3, 1, 4) OVER (ORDER BY t) FROM public.my_time_series_table ) ) OPTIONS ( 'numVariables' -> '3', 'numLags' -> '4' ); ``` **Prepare Data for a VAR Model with Manual Vector Assembly** This version of the univariate regression example uses repeated `LAG` invocations instead of the `LAGS` function. Although it produces the exact same set of lagged features (`sales_lag1`, `sales_lag2`, `sales_lag3`), the query requires writing each `LAG` expression manually. This makes the SQL longer, more repetitive, and harder to maintain compared to the concise single-line `LAGS` version. ```sql SQL theme={null} CREATE MLMODEL my_var_model_manual TYPE VECTOR_AUTOREGRESSION ON ( SELECT {x1, x2, x3}, {x1_lag1, x2_lag1, x3_lag1}, {x1_lag2, x2_lag2, x3_lag2}, {x1_lag3, x2_lag3, x3_lag3}, {x1_lag4, x2_lag4, x3_lag4} FROM ( SELECT x1, x2, x3, COALESCE(LAG(x1, 1) OVER (ORDER BY t), 0) AS x1_lag1, COALESCE(LAG(x1, 2) OVER (ORDER BY t), 0) AS x1_lag2, COALESCE(LAG(x1, 3) OVER (ORDER BY t), 0) AS x1_lag3, COALESCE(LAG(x1, 4) OVER (ORDER BY t), 0) AS x1_lag4, COALESCE(LAG(x2, 1) OVER (ORDER BY t), 0) AS x2_lag1, COALESCE(LAG(x2, 2) OVER (ORDER BY t), 0) AS x2_lag2, COALESCE(LAG(x2, 3) OVER (ORDER BY t), 0) AS x2_lag3, COALESCE(LAG(x2, 4) OVER (ORDER BY t), 0) AS x2_lag4, COALESCE(LAG(x3, 1) OVER (ORDER BY t), 0) AS x3_lag1, COALESCE(LAG(x3, 2) OVER (ORDER BY t), 0) AS x3_lag2, COALESCE(LAG(x3, 3) OVER (ORDER BY t), 0) AS x3_lag3, COALESCE(LAG(x3, 4) OVER (ORDER BY t), 0) AS x3_lag4 FROM public.my_time_series_table ) ) OPTIONS ( 'numVariables' -> '3', 'numLags' -> '4' ); ``` ## Related Links [Window Aggregate Functions](/window-aggregate-functions) [Machine Learning in Ocient](/machine-learning-in-ocient) [Machine Learning Models](/machine-learning-models) # Data Query Language (DQL) Statement Reference Source: https://docs.ocient.com/data-query-language-dql-statement-reference Explore Ocient SQL syntax rules, enabling users to write clear, efficient queries for data retrieval and manipulation at scale. The supports querying using the SQL syntax that follows ANSI SQL standards. ## Ocient SQL Syntax This code block shows the general syntax to perform SQL querying in and the order in which commands should go. For specific descriptions and syntax for the commands, see the respective SQL statement sections on this page. **Syntax** ```sql SQL theme={null} [ WITH ... ] SELECT ... [ EXCEPT(...) ] [ FROM ... [ JOIN ... ] ] [ WHERE ... ] [ GROUP BY ... [ HAVING ... ] ] [ ORDER BY ... ] [ LIMIT ... ] [ OFFSET ... ] [ INTERSECT ... ] [ EXCEPT ... ] [ UNION ... ] ] [ USING ... ] [ TRACE ... ] [ TAG ... ] ``` ## Default Schema Ocient identifies every table using a database and schema. For example, the fully qualified path to the `movies` table is `cinema.adventure.movies`, where `cinema` is the database and `adventure` is the schema. When you do not fully qualify a table name, the Ocient System uses a default schema. When you first log into the system, the default schema is your fully qualified username. You can change the default schema using the [SET SCHEMA](/commands-supported-by-the-ocient-jdbc-cli-program#set-schema) command. ## Querying SQL Statement Reference Ocient supports the following SQL statements. ### WITH Assigns a name to a common table expression, allowing an auxiliary query to be used in the main query. This helps break complex queries into smaller parts. **Syntax** ```sql SQL theme={null} WITH [ ( [ ,... ] ) ] AS ( ) ``` **Parameters** | **Parameter** | **Description** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `` | A name for the common table expression populated with the result set from the `sub_query` statement. | | `` | Optional.
This is a list of column names for common table expressions populated with the `sub_query` result set. If no column names are provided, the column names are the same as the table referenced in the `sub_query` statement. | | `` | An auxiliary query that collects data for a common table expression.
`sub_query` can use regular query commands, including ORDER BY, LIMIT, OFFSET, UNION, INTERSECT, EXCEPT, WHERE, GROUP BY, and HAVING.
See the respective SQL statement reference sections for usage rules. | | `` | The main SELECT query.
Note that the `FROM` statement in the main `SELECT` query must also include the `cte_name` if the query uses its values. | **Example** In this example, the subquery calculates the average budget for all rows in the `movies` table. The main query uses that average to find all movies that spent more. ```sql SQL theme={null} WITH avg_budget_table (average_budget) AS ( SELECT AVG(budget) FROM movies ) SELECT title, budget, revenue FROM movies, avg_budget_table WHERE budget > avg_budget_table.average_budget; ``` *Output* | title | budget | revenue | | ----------------------- | --------- | ---------- | | CGI Why | 237000000 | 2787965087 | | Titania | 200000000 | 1845034188 | | Merchandise Vehicle 5 | 200000000 | 1066969703 | | The Tentpole | 220000000 | 1519557910 | | Pirates of Palm Springs | 140000000 | 655011224 | | Spyman 16 | 200000000 | 1108561013 | | Frigid | 150000000 | 1274219009 | | Fury 7 | 190000000 | 1506249360 | | Superhero 23 | 250000000 | 1084939099 | | Triassic World | 150000000 | 1513528810 | | Iron Chef 3 | 200000000 | 1215439994 | | The Last Airman | 150000000 | 318502923 | ### SELECT Initiates a query statement or a subquery clause within other statements. You can query the data of tables where you have the SELECT privilege. For information on using SELECT as a subquery for filtering or ordering results, see the [WHERE](#where) and [HAVING](#having) sections. For information on using SELECT as a subquery for a common table expression, see the [WITH](#with) section. `SELECT *` queries that lack a `FROM` clause automatically reference the `sys.dummy1` table. For example, `SELECT *;` is the same as `SELECT * FROM sys.dummy1;`. **Syntax** ```sql SQL theme={null} SELECT [ ALL | DISTINCT ] [ * [ EXCEPT ( column_name [ , ... ] ) ] | [ , ... ] ] [ ] ``` **Parameters** | **Parameter** | Description | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `ALL` | `SELECT` and `SELECT ALL` are both the same.
Returns all valid data rows from the database that meet the criteria of your query. | | `DISTINCT` | `SELECT DISTINCT` returns only unique rows that do not match other rows based on the criteria of your query. | | `*` | Returns all columns from the specified tables in the result set for the query. | | `EXCEPT` | When used with `*`, `EXCEPT` allows specific columns to be excluded from the query results. | | `column_name` | The name of one or more columns from the specified table that you want to exclude (using `EXCEPT`) from your query results. | #### **\** The `select_list_entry` defines a column or expression to include in your query result set. **Syntax** ```sql SQL theme={null} ::= column_name | expression [ AS new_name ] ``` **Parameters** | **Parameter** | **Description** | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `column_name` | The name of one or more columns from the specified table that you want to include in your query results. | | `expression` | One or more expressions that you want to include in your query results.
Expressions can be any combination of literal values, column names, arithmetic expressions, parentheses, and function calls. | | `new_name` | An alias, treated as an identifier, for an alternative name for the column or the results of the expression. | #### **\** For information on the ``, see the [FROM](#from) documentation. **Examples** #### Using SELECT \* This example uses `SELECT *` to return all the columns in the `movies` table. ```sql SQL theme={null} SELECT * FROM movies LIMIT 5; ``` *Output* | movie\_id | title | budget | popularity | release\_date | revenue | runtime | movie\_status | vote\_average | vote\_count | | --------- | --------------- | --------- | ---------- | ------------- | ---------- | ------- | ------------- | ------------- | ----------- | | 211672 | Minnows | 74000000 | 875.581305 | 2015-06-17 | 1156730962 | 91 | Released | 6.40 | 4571 | | 24 | Billy the Killy | 30000000 | 79.754966 | 2003-10-10 | 180949000 | 111 | Released | 7.70 | 4949 | | 19995 | CGI Why | 237000000 | 150.437577 | 2009-12-10 | 2787965087 | 162 | Released | 7.20 | 11800 | | 37724 | Spyman 16 | 200000000 | 93.004993 | 2012-10-25 | 1108561013 | 143 | Released | 6.90 | 7604 | | 24428 | The Tentpole | 220000000 | 144.448633 | 2012-04-25 | 1519557910 | 143 | Released | 7.40 | 11776 | #### Using SELECT \* EXCEPT This example uses `SELECT * EXCEPT` to exclude certain columns from the result set. ```sql SQL theme={null} SELECT * EXCEPT (movie_id, runtime, vote_average, vote_count) FROM movies LIMIT 5; ``` *Output* | title | budget | release\_date | revenue | | --------------------- | --------- | ------------- | ---------- | | Swords & Scabbards | 94000000 | 2003-12-01 | 1118888979 | | Billy the Killy | 30000000 | 2003-10-10 | 180949000 | | Merchandise Vehicle 5 | 200000000 | 2010-06-16 | 1066969703 | | CGI Why | 237000000 | 2009-12-10 | 2787965087 | | Space Odyssey 6000 | 10500000 | 1968-04-10 | 68700000 | #### Using Lateral Column Aliases Ocient SQL queries support lateral column aliases, meaning you can immediately reuse aliases for calculations in the same query as new inputs. Hence, you can simplify queries that normally require subqueries and common table expressions. These examples use the `products` table with these columns: * `product_id` — Product identifier as an integer * `product_name` — Product name as a string * `price` — Price as a floating point number Create this table using the `CREATE TABLE` SQL statement. ```sql SQL theme={null} CREATE TABLE products ( product_id INT, product_name VARCHAR(100), price DECIMAL(10, 2) ); ``` Insert four records into the `products` table. ```sql SQL theme={null} INSERT INTO products (product_id, product_name, price) VALUES (1, 'Laptop', 1000.00), (2, 'Tablet', 500.00), (3, 'Smartphone', 800.00), (4, 'Monitor', 300.00); ``` Create a query to determine prices after discounts and taxes by using a common table expression subquery `DiscountedPrices`. Use the `WITH` keyword to create the subquery. ```sql SQL theme={null} WITH DiscountedPrices AS ( SELECT product_id, product_name, price, price * 0.90 AS discounted_price, price * 0.90 * 1.05 AS total_price_after_tax FROM products ) SELECT product_id, product_name, price, discounted_price, total_price_after_tax FROM DiscountedPrices; ``` Lateral aliases allow the same calculations to be packaged in a single query. This simpler query is essentially the same as the longer common table expression example, but the logic is condensed because you can reference the `discounted_price` alias immediately to calculate the `total_price_after_tax` value in the same query. ```sql SQL theme={null} SELECT product_id, product_name, price, price * 0.90 AS discounted_price, discounted_price * 1.05 AS total_price_after_tax FROM products; ``` ### FROM Specifies the table or view to use in a `SELECT` statement. **Syntax** ```sql SQL theme={null} FROM { table_name | ( ) | ( ) } [ ,... ] [ [ ,... ] ] ``` **Parameters** | **Parameter** | Description | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | A query statement using the `SELECT` SQL statement. For details, see the [SELECT](#select) section. | | `table_name` | The names of one or more tables or views that you want to query.
If you specify multiple tables in the `FROM` list, separated by commas, the effect is the same as if you explicitly perform a [CROSS JOIN](#cross-join) on all of those sources. | | `` | One or more subqueries using the `SELECT` SQL statement. A subquery generates a table from which the `select_clause` query references data.
Each subquery must be enclosed in parentheses with an optional correlation name.
For details, see the [SELECT](#select) section. | | `` | Use the VALUES keyword to define a table with data. For example, the `SELECT * FROM (VALUES (1, NULL), (2, 5))` SQL statement selects all the data from a table defined by the specified values as `(1, NULL)` for the first row with two columns, and `(2, 5)` for the second row. | | `` | A `JOIN` clause used to retrieve data from two or more tables for your query.
For details, see the [JOIN](#join) section. | ### JOIN Combines rows from multiple tables so they can be accessed by a query. **Syntax** ```sql SQL theme={null} FROM ON table1_column table2_column ``` **Parameter** | **Parameter** | **Description** | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | A valid `SELECT` query.
For details, see [SELECT](#select). | | `` | Either a table name or a full `SELECT` in parentheses
`table_reference1` takes precedence for returning rows for `LEFT OUTER JOIN`, `SEMI JOIN`, and `ANTI JOIN`. For specific rules, see [Types of Join Operations](#types-of-join-operations). | | `` | Either a table name or a full `SELECT` in parentheses function.
`table_reference2` takes precedence for returning rows for `RIGHT OUTER JOIN`. For specific rules, see [Types of Join Operations](#types-of-join-operations). | | `table1_column` | A column from either `table_reference`
The `JOIN` operation uses this column to match rows with `table2_column` to combine data for the result set. | | `` | Join conditions can use any Boolean expression, including `=`, `!=`, `<`, `>`, `=>`, and `<=`. | | `table2_column` | A column from the `table_reference` not used for `table1_column`
The `JOIN` operation uses this column to match rows based on the `table1_column` to combine data for the result set. | #### Types of Join Operations ( `` ) \[#types-of-join-operations] Ocient supports the following types of `JOIN` operations. **Syntax** ```sql SQL theme={null} ::= { INNER JOIN | LEFT [ OUTER ] JOIN | RIGHT [ OUTER ] JOIN | FULL [ OUTER ] JOIN | CROSS JOIN | SEMI JOIN | ANTI JOIN } ``` #### **JOIN Type Descriptions** | **JOIN Type** | **Description** | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `INNER JOIN` | Returns all rows that exist in both tables. This join type is the default type of join. | | `LEFT OUTER JOIN` | Returns all rows from the first table regardless of whether there are matching rows in the second table.
`LEFT JOIN` performs the same operation as `LEFT OUTER JOIN`. | | `RIGHT OUTER JOIN` | Returns all rows from the second table regardless of whether there are matching rows in the first table.
`RIGHT JOIN` performs the same operation as `RIGHT OUTER JOIN`. | | `FULL OUTER JOIN` | Returns all matched and unmatched rows.
`FULL JOIN` performs the same operation as `FULL OUTER JOIN`. | | `CROSS JOIN` | Returns the Cartesian product of the joined tables. Each row in the first table is joined with all rows in the second table. The result set contains the number of rows in the first table multiplied by the number of rows in the second table. | | `SEMI JOIN` | Returns the rows from the first table that have at least one match in the second table. No columns from the second table are available in the query. | | `ANTI JOIN` | Returns the rows from the first table that have no matches in the second table. No columns from the second table are available in the query. | By default, `JOIN` statements that involve subqueries can operate laterally if necessary. This behavior allows subqueries to reference joined columns from the preceding items included in the `FROM` clause. For example, both parts of this join operation use subqueries that reference table `x`. The `LATERAL` keyword is optional; the join operation acts the same regardless of whether it is included. ```text Text theme={null} SELECT \* FROM ( SELECT 1 AS cx1, 2 AS cx2) AS x INNER JOIN LATERAL ( SELECT x.cx1 AS cy1, x.cx2 AS cy2) AS y ON x.cx1 = y.cy1; ``` Lateral joins are primarily useful when a cross-referenced column is necessary for computing the rows to join. A common application is providing an argument value for a set-returning function. **Examples** These examples join two tables: * `games` — A table of video game titles and their genre\_ids. Note that some games have a NULL value assigned to their genre\_id. | game\_name | genre\_id | | ---------------------- | --------- | | Star Battle 6000 | 9 | | Dwarf Simulator | NULL | | 2002 Futbol | 11 | | Space Attack! | 9 | | Fantasy Fool | 8 | | Blasto the Squirrel | 5 | | Trucker Quest | 7 | | Block, Stack, and Cry! | 6 | | Basketball Day | 11 | | Pizza Mutant | 1 | | Italian Plumber | 5 | | Zombies! | 1 | | Space Ninjas | 12 | | Moon Mayor | 10 | | Spreadsheet Hero | NULL | | Zoom Tycoon | 12 | | Porch Poacher | 6 | * `genre` — A table of video game genres, which are identified by IDs. | genre\_name | id | | ------------ | -- | | Racing | 7 | | Puzzle | 6 | | Adventure | 2 | | Simulation | 10 | | Shooter | 9 | | Misc | 4 | | Action | 1 | | Sports | 11 | | Role-Playing | 8 | | Platform | 5 | | Strategy | 12 | | Fighting | 3 | #### INNER JOIN This example uses an `INNER JOIN` operation to capture only the rows that exist in both tables. Games with NULL values for their `genre_id` are eliminated from the result set. ```sql SQL theme={null} SELECT game_name, genre_name FROM video_games.game INNER JOIN video_games.genre ON game.genre_id = genre.id; ``` *Output* | game\_name | genre\_name | | ---------------------- | ------------ | | Fantasy Fool | Role-Playing | | Blasto the Squirrel | Platform | | Italian Plumber | Platform | | Porch Poacher | Puzzle | | Space Ninjas | Strategy | | Basketball Day | Sports | | 2002 Futbol | Sports | | Moon Mayor | Simulation | | Block, Stack, and Cry! | Puzzle | | Zombies! | Action | | Pizza Mutant | Action | | Space Attack! | Shooter | | Trucker Quest | Racing | | Zoom Tycoon | Strategy | #### LEFT OUTER JOIN This example uses a `LEFT OUTER JOIN` operation to capture all rows from the left table (game), even if they have no matching row in the right table (genre). ```sql SQL theme={null} SELECT game_name, genre_name FROM video_games.game LEFT OUTER JOIN video_games.genre ON game.genre_id = genre.id; ``` *Output* | game\_name | genre\_name | | ---------------------- | ------------ | | Fantasy Fool | Role-Playing | | Space Ninjas | Strategy | | Zoom Tycoon | Strategy | | Zombies! | Action | | Pizza Mutant | Action | | Trucker Quest | Racing | | Blasto the Squirrel | Platform | | 2002 Futbol | Sports | | Basketball Day | Sports | | Italian Plumber | Platform | | Block, Stack, and Cry! | Puzzle | | Star Battle 6000 | Shooter | | Dwarf Simulator | NULL | | Moon Mayor | Simulation | | Porch Poacher | Puzzle | | Spreadsheet Hero | NULL | | Space Attack! | Shooter | #### RIGHT OUTER JOIN This example uses a `RIGHT`` OUTER JOIN` operation to capture all rows from the right table (genre), even if they have no matching row in the left table (game). ```sql SQL theme={null} SELECT game_name, genre_name FROM video_games.game RIGHT OUTER JOIN video_games.genre ON game.genre_id = genre.id; ``` *Output* | game\_name | genre\_name | | ---------------------- | ------------ | | NULL | Adventure | | Trucker Quest | Racing | | Pizza Mutant | Action | | Moon Mayor | Simulation | | Fantasy Fool | Role-Playing | | Space Attack! | Shooter | | NULL | Misc | | Star Battle 6000 | Shooter | | Space Ninjas | Strategy | | Porch Poacher | Puzzle | | Block, Stack, and Cry! | Puzzle | | Blasto the Squirrel | Platform | | Italian Plumber | Platform | | Zoom Tycoon | Strategy | | NULL | Fighting | | Zombies! | Action | | 2002 Futbol | Sports | | Basketball Day | Sports | #### FULL OUTER JOIN This example uses a `FULL OUTER JOIN` operation to capture all rows from both tables, even if they do not match. ```sql SQL theme={null} SELECT game_name, genre_name FROM video_games.game FULL OUTER JOIN video_games.genre ON game.genre_id = genre.id; ``` *Output* | game\_name | genre\_name | | ---------------------- | ------------ | | Basketball Day | Sports | | 2002 Futbol | Sports | | Zombies! | Action | | Blasto the Squirrel | Platform | | Moon Mayor | Simulation | | Trucker Quest | Racing | | NULL | Misc | | Star Battle 6000 | Shooter | | Space Attack! | Shooter | | Spreadsheet Hero | NULL | | NULL | Adventure | | NULL | Fighting | | Italian Plumber | Platform | | Pizza Mutant | Action | | Dwarf Simulator | NULL | | Fantasy Fool | Role-Playing | | Block, Stack, and Cry! | Puzzle | | Space Ninjas | Strategy | | Zoom Tycoon | Strategy | | Porch Poacher | Puzzle | #### CROSS JOIN This example uses a `CROSS JOIN` operation to capture every possible combination of rows from both tables, regardless of whether they match. Note that, unlike other `JOIN` operations, `CROSS JOIN` does not require an `ON` statement. ```sql SQL theme={null} SELECT game_name, genre_name FROM video_games.genre CROSS JOIN video_games.game; ``` *Output* As the result set for this CROSS JOIN example is more than 200 rows, the results are abbreviated. | genre\_name | genre\_id | | ---------------- | ------------ | | Spreadsheet Hero | Role-Playing | | Spreadsheet Hero | Misc | | Spreadsheet Hero | Sports | | Spreadsheet Hero | Platform | | Spreadsheet Hero | Strategy | | Spreadsheet Hero | Shooter | | Spreadsheet Hero | Fighting | | Spreadsheet Hero | Action | | Spreadsheet Hero | Simulation | | Spreadsheet Hero | Racing | | Spreadsheet Hero | Puzzle | | Spreadsheet Hero | Adventure | | Space Attack! | Role-Playing | | Space Attack! | Misc | | Space Attack! | Sports | | Space Attack! | Platform | | Space Attack! | Strategy | | Space Attack! | Shooter | | Space Attack! | Fighting | | Space Attack! | Action | | Space Attack! | Simulation | | Space Attack! | Racing | | Space Attack! | Puzzle | | Space Attack! | Adventure | | ... | ... | #### SEMI JOIN This example uses a `SEMI JOIN` operation to capture genre names that have at least one match in the games table. Rows are only included once, even if there are multiple matches. ```sql SQL theme={null} SELECT genre_name, id FROM video_games.genre SEMI JOIN video_games.game ON genre.id = game.genre_id; ``` *Output* | genre\_name | genre\_id | | ------------ | --------- | | Simulation | 10 | | Shooter | 9 | | Racing | 7 | | Strategy | 12 | | Puzzle | 6 | | Platform | 5 | | Action | 1 | | Sports | 11 | | Role-Playing | 8 | #### ANTI JOIN This example uses an `ANTI JOIN` operation to capture any rows in the game table that do not match any rows in the genre table. ```sql SQL theme={null} SELECT game_name FROM video_games.game ANTI JOIN video_games.genre ON game.genre_id = genre.id; ``` *Output* | game\_name | | ---------------- | | Spreadsheet Hero | | Dwarf Simulator | ### WHERE Filters rows based on a specified condition. **Syntax** ```sql SQL theme={null} WHERE { column_name filter_value | [ NOT ] EXISTS ( ) } ``` **Parameters** | **Parameter** | **Description** | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `` | A valid `SELECT` query. | | `column_name` | A column used for grouping that is evaluated by the `filter_condition`. | | `filter_value` | A value used to evaluate the specified `column_name`.
For details, see the [\](#filtercondition) section. | | `` | A subquery with a filter condition that follows the `EXISTS` clause.
For details, see the [EXISTS](#exists) section. | #### \ A logical combination of predicates used to evaluate the referenced `column_name` based on the `filter_value`. **Syntax** ```sql SQL theme={null} ::= { = | == | <> | != | [ NOT ] EQUALS | < | <= | > | >= | [ NOT ] IN | [ NOT ] LIKE | [ NOT ] SIMILAR TO | BETWEEN | FOR SOME | FOR ALL | IS [ NOT ] NULL | IS [ NOT ] DISTINCT FROM } ``` **Definitions** | **Operator** | **Description** | **Example** | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | `=`, `==`, `EQUALS` | Equal to. | `Author = 'Alcott'` | | `<>`, `!=` | Not equal to | `Dept <> 'Sales'` | | `>` | Greater than | `Hire_Date > '2012-01-31'` | | `<` | Less than | `Bonus < 50000.00` | | `>=` | Greater than or equal | `Dependents >= 2` | | `<=` | Less than or equal | `Rate <= 0.05` | | `[ NOT ] LIKE` | Begins with a character pattern to match.
The pattern is enclosed in parentheses and can include the wildcard characters `%`, representing zero, one or multiple characters, and `_` representing a single character.
To have a literal `%` or `_` that does not act as a wildcard on the right-hand side of a `LIKE` or `NOT LIKE` expression, put a backslash before the character. | `Full_Name LIKE 'Will%'` | | `[ NOT ] SIMILAR TO` | For usage information, see the [SIMILAR TO Operator](#similar-to-operator) section. | `'abc' SIMILAR TO 'abc' ` | | `BETWEEN value1 AND value2` | Requires two values.
The filter evaluates to TRUE if the specified column is within the range between the two values. `BETWEEN` bounds are inclusive. Values can be times, numbers, or strings. | `Sales BETWEEN 1000 AND 5000` | | `FOR_SOME()` | Evaluates an array to determine if at least one value meets the filter criteria.
In the example, the `FOR_SOME()` operator would return all rows of the column `col_int_array` where at least one value is greater than 10.
For details, see [Array Filters](#array-filters). | `FOR_SOME(col_int_array) > 10`

`FOR_SOME(col_int_array) > 10` | | `FOR_ALL()` | Evaluates all values of an array to determine if all meet the filter criteria.
In the example, the `FOR_ALL()` operator would return any rows of the column `col_int_array` where all values are greater than 10.
For details, see [Array Filters](#array-filters). | `FOR_ALL(col_int_array) > 10`

`FOR_ALL(col_int_array) = 1` | | `[ NOT ] IN` | Equal to one of multiple possible values | `DeptCode IN (101, 103, 209)` | | `IS [ NOT ] NULL` | Compare to `NULL` (missing data) | `Address IS NOT NULL` | | `IS [ NOT ] DISTINCT FROM` | Compares the equality of two expressions for a Boolean result, including comparisons that involve `NULL` values. This operator ensures reliable comparisons in scenarios where `NULL` values need to be treated as significant data points.
Normally, `NULL = NULL` evaluates to `false` because `NULL` represents an unknown value. However, the statement `a IS NOT DISTINCT FROM b` treats `NULL` as a comparable value and returns `true` when both values are `NULL` or when `a = b`.
Inversely, `a IS DISTINCT FROM b` returns `true` if the values are different or if one value is `NULL` while the other is not `NULL`. | `Debt IS NOT DISTINCT FROM Receivables` | #### EXISTS An `EXISTS` clause used in a `WHERE` statement evaluates if a subquery returns any rows. For each row that the database computes in the outer query, if the subquery returns at least one row, the `EXISTS` clause evaluates to `true`. In this outcome, the outer query returns its row. If the subquery returns zero rows, the `EXISTS` clause evaluates to `false`, and the outer query excludes those rows. The `NOT EXISTS` clause performs the opposite Boolean logic. In this case, the outer query returns rows only when the subquery has no matches. **Examples** These examples use two tables, one for company departments and the other for employees assigned to those departments. Create the `departments` table for department data. ```sql SQL theme={null} CREATE TABLE departments ( id INT, name VARCHAR(50) ); ``` Create the `employees` table for employee data. ```sql SQL theme={null} CREATE TABLE employees ( id INT, name VARCHAR(50), department_id INT ); ``` Insert department data for three departments, `HR`, `IT`, and `Marketing`, into the `departments` table. ```sql SQL theme={null} INSERT INTO departments (id, name) VALUES (1, 'HR'), (2, 'IT'), (3, 'Marketing'); ``` Insert employee data for four employees, `Alice`, `Bob`, `Charlie`, and `David`, into the `employees` table. ```sql SQL theme={null} INSERT INTO employees (id, name, department_id) VALUES (1, 'Alice', 1), (2, 'Bob', 2), (3, 'Charlie', 2), (4, 'David', 4); ``` **Find Employees Belonging to an Existing Department** This example uses an `EXISTS` clause to find any names from the `employees` table who are assigned to a department listed in the `departments` table. ```sql SQL theme={null} SELECT name FROM employees AS e WHERE EXISTS ( SELECT * FROM departments AS d WHERE e.department_id = d.id ); ``` The query returns all employees except `David`, who is assigned to a department that does not exist in the `departments` table. *Output* ```none Text theme={null} Bob Charlie Alice ``` **Find Departments That Have Employees** This query returns departments that have at least one employee. The subquery uses `SELECT 1` to check whether any matching rows exist in the `employees` table. The output is the same if the query uses `SELECT *` instead. ```sql SQL theme={null} SELECT name FROM departments AS d WHERE EXISTS ( SELECT 1 FROM employees AS e WHERE e.department_id = d.id ); ``` The query results exclude the `Marketing` department because no employees belong to it. *Output* ```none Text theme={null} HR IT ``` **Filter Departments Based on an Uncorrelated Table** In this example, the outer query filters the `departments` table where `id != 3` (excluding the `Marketing` department). The subquery checks if there is at least one employee with a department identifier less than `4`. The query is uncorrelated because the subquery does not reference the `departments` table. ```sql SQL theme={null} SELECT * FROM departments d WHERE d.id != 3 AND EXISTS ( SELECT * FROM employees e WHERE e.department_id < 4 ); ``` *Output* ```none Text theme={null} HR IT ``` **Find Employees Without a Valid Department** This example uses a `NOT EXISTS` clause. The example returns only employees who are assigned to a department identifier `department_id` not listed in the `departments` table. ```sql SQL theme={null} SELECT name FROM employees AS e WHERE NOT EXISTS ( SELECT * FROM departments AS d WHERE e.department_id = d.id ); ``` \*Output: \*`David` #### SIMILAR TO Operator `SIMILAR TO` is a keyword that extends the `LIKE` operator, adding more features for match filtering, including many metacharacters used in regular expressions. `%` and `_` both act as wildcard operators, but other supported metacharacters match traditional regular expressions. **Syntax** ```sql SQL theme={null} WHERE string1 SIMILAR TO string2 ``` This table describes the metacharacters supported by the `SIMILAR TO` keyword. | **Metacharacter** | **Description** | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `%` | Repeated wildcard, matches any characters zero or more times. Equivalent usage to `LIKE`. | | `_` | Wildcard, matches exactly one character. Equivalent usage to `LIKE`. | | `\|` | Alteration, meaning either of two alternatives (`a\|b` represents `a` or `b`). | | `*` | Repetition of zero or more times. | | `+` | Repetition of one or more times. | | `?` | Repetition of zero or one times. | | `{m}` | Repetition of exactly `m` times. | | `{m,n}` | Repetition between `m` and `n` times (inclusive). | | `{m,}` | Repetition of at least `m` times. | | `()` | Logical grouping. | | `[]` | A character class, equivalent to character classes in regular expressions. Example: `[a-z]` is any lowercase English letter. | | `^` | Beginning of the line anchor and the negation in character classes. | | `$` | End of the line anchor. | `SIMILAR TO` also supports escaping these metacharacters with `\` and certain escape sequences supported by C family languages. | **Metacharacter** | **Description** | | ------------------- | ------------------------------------------------------------------------------- | | `\a` | The alert/bell character. | | `\b` | The backspace character. | | `\B` | A single `\` character. Equivalent to `\\`. | | `\f` | Form feed. | | `\t` | Horizontal tab. | | `\v` | Vertical tab. | | `\xy` | `xy` are octal digits that represent the character with the numeric value `xy`. | | `\s` | Matches any whitespace. | | `\S` | Matches any non-whitespace. | | `\m`, `\M`, or `\y` | The boundary of a word. Equivalent to `\b` in normal regular expressions. | | `*` | The star character. | #### Array Filters A filter expression can use the functions `FOR_SOME()` and `FOR_ALL()` to apply a predicate against all values of an input array. Array filter functions have the following rules: * They can only evaluate array types. * They can go on either the left or right side of a Boolean comparison expression, but not both sides at the same time. * They must directly use a Boolean comparison operator. * They cannot use the `SOME` and `ALL` SQL keywords. **Filter Behavior with Empty Arrays** Array filter functions have unique behavior when evaluating empty arrays. * `FOR_ALL()` evaluates an empty array as `TRUE`. * `FOR_SOME()` evaluates an empty array as `FALSE`. If empty arrays must be evaluated for a different result, you can use the `ARRAY_LENGTH` function to specify a minimum array length. For example, this statement would evaluate an array as `TRUE` only if it is not empty and all values matched `%ocient%`: ```sql SQL theme={null} ARRAY_LENGTH(array_col) > 0 AND FOR_ALL(array_col) LIKE '%ocient% ``` For details about array functions, see the [Array Functions and Operators](/array-functions-and-operators) page. **Filter Behavior with NULL rows** Filtering with array functions can yield different results depending on whether the array row is NULL or whether the values in the array are NULL. If `FOR_SOME()` or `FOR_ALL()` evaluates a NULL row (the row itself is NULL, not that the array contains NULL values), then the result is always `FALSE`. To check an array for the presence of NULL values, you must use the `IS NULL` operator. For example: ```sql SQL theme={null} FOR_SOME(array_col) IS NULL ``` Array comparison operators, such as `@>`, `<@`, and `&&`, do not adhere to Boolean logic for NULL values. For details, see the [Array Functions and Operators](/array-functions-and-operators) page. ### GROUP BY Groups rows with the same values into summary rows, based on a specified aggregate function. **Syntax** ```sql SQL theme={null} SELECT FROM table_name GROUP BY { column_name | expression | integer } [ , ... ] [ HAVING ] ``` **Parameters** | **Parameter** | **Description** | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | A list of one or more columns for the `SELECT` query.
To use `GROUP BY` to summarize the query result set, at least one of the columns in `column_list` must include an aggregate function, such as `SUM()`, `COUNT()`, or `MAX()`.
For a list of supported functions, see the [Aggregate Functions](/aggregate-functions) page. | | `table_name` | The table to use for the query. | | `column_name` | The name of one or more columns to use to group the result set.
If you specify multiple columns, the result set is grouped by each unique combination of values from the columns. | | `expression` | Any combination of literal values, column names, arithmetic expressions, parentheses, and function calls. | | `integer` | An integer representing the position of the columns referenced in the `column_list`. The first position starts at `1`. | | `` | A `HAVING` clause that filters the aggregated groups based on a specified condition.
For details, see the [\](#filtercondition) section. | **Example** This example uses a movie database to calculate the total amount spent on movie production per year. ```sql SQL theme={null} SELECT YEAR(release_date), SUM(budget) FROM movies GROUP BY 1 ORDER BY 1 ASC; ``` *Output* | year(release\_date) | sum(budget) | | ------------------- | ----------- | | 1968 | 10500000 | | 1979 | 31500000 | | 1981 | 18000000 | | 1982 | 28000000 | | 1992 | 14000000 | | 1997 | 200000000 | | 2003 | 264000000 | | 2009 | 237000000 | | 2010 | 350000000 | | 2012 | 670000000 | | 2013 | 350000000 | | 2015 | 414000000 | ### HAVING Filters aggregated rows based on a specified condition. `HAVING` operates in a `GROUP BY` statement by setting a filter for the rows to be aggregated and grouped. For information on using `GROUP BY` in a query statement, see [GROUP BY](#group-by). **Syntax** ```sql SQL theme={null} GROUP BY column_name HAVING ``` **Parameters** | **Parameter** | **Description** | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `column_name` | A column used for grouping that is evaluated by the `filter_condition`. | | `` | A logical combination of Boolean predicates.
For information on supported logical predicates, see [\](#filtercondition). | **Example** This example calculates the total amount spent on movie production per year. The `HAVING` clause filters out any rows that do not have a sum of at least \$100 million. ```sql SQL theme={null} SELECT YEAR(release_date), SUM(budget) FROM movies GROUP BY 1 HAVING SUM(budget) > 100000000 ORDER BY 1 ASC; ``` *Output* | year(release\_date) | sum(budget) | | ------------------- | ----------- | | 1997 | 200000000 | | 2003 | 264000000 | | 2009 | 237000000 | | 2010 | 350000000 | | 2012 | 670000000 | | 2013 | 350000000 | | 2015 | 414000000 | ### ORDER BY Sorts the result set in ascending or descending order based on one or more specified columns. If you specify multiple columns, they are sorted hierarchically from left to right. **Syntax** ```sql SQL theme={null} ORDER BY { column_position | column_name } [ ASC | DESC ] [ NULLS FIRST | NULLS LAST ] [ , ... ] ``` **Parameters** | **Parameter** | **Data Type** | **Description** | | --------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `column_position` | Integer | The position of a column to be used for the `ORDER BY` sorting.
Column positions start at `1`. | | `column_name` | String | The name of a column to be used for the `ORDER BY` sorting. | | `ASC \| DESC` | String | Optional.
Specifies whether to sort the column in ascending (`ASC`) or descending (`DESC`) order.
If unspecified, defaults to `ASC`. | | `NULLS FIRST \| NULLS LAST` | String | Optional.
`NULLS FIRST` means NULL values go at the start of the result set.
`NULLS LAST` means NULL values go at the end of the result set.

If unspecified, defaults to `NULLS FIRST`. | **Example** In this example, the `ORDER BY` statement orders the movies from newest to oldest. ```sql SQL theme={null} SELECT release_date, title FROM movies ORDER BY release_date DESC; ``` *Output* | release\_date | title | | ------------- | ---------------------------- | | 2015-06-17 | Minnows | | 2015-06-09 | Triassic World | | 2015-04-01 | Fury 7 | | 2013-11-27 | Frigid | | 2013-04-18 | Iron Chef 3 | | 2012-10-25 | Spyman 16 | | 2012-07-16 | Superhero 23 | | 2012-04-25 | The Tentpole | | 2010-06-30 | The Last Airman | | 2010-06-16 | Merchandise Vehicle 5 | | 2009-12-10 | CGI Why | | 2003-12-01 | Swords & Scabbards | | 2003-10-10 | Billy the Killy | | 2003-07-09 | Pirates of Palm Springs | | 1997-11-18 | Titania | | 1992-08-07 | Bang Bang Western | | 1982-06-25 | Blade Walker | | 1981-06-12 | Raiders of the Jungle Temple | | 1979-08-15 | Apocalypse None | | 1968-04-10 | Space Odyssey 6000 | ### LIMIT Limits the number of returned rows from a query to a specified amount. The returned rows are nondeterministic unless you use an [ORDER BY](#order-by) clause. **Syntax** ```sql SQL theme={null} LIMIT limit_number ``` **Parameters** | **Parameter** | **Description** | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | A valid `SELECT` query. | | `limit_number` | The number of rows to return from the query, specified as a positive integer or any constant scalar expression that evaluates to a positive integer. | **Examples** **Limit Returned Rows Using a Number** This example uses `LIMIT` to restrict the number of returned movie titles to only three. ```sql SQL theme={null} SELECT title FROM movies LIMIT 3; ``` *Output* ```sql SQL theme={null} Minnows CGI Why Billy the Killy ``` **Limit Returned Rows Using an Expression** The `LIMIT` SQL statement can also accept expressions. This query uses the `1+2` expression with the `sys.dummy` table to create a table of three incrementing integers. For details, see [Generate Tables Using sys.dummy](/generate-tables-using-sys-dummy). ```sql SQL theme={null} SELECT c1 FROM sys.dummy10 LIMIT 1+2; ``` *Output* ```sql SQL theme={null} 1 2 3 ``` ### OFFSET Skips a specified number of rows from the result set. The returned rows are nondeterministic unless you use an [ORDER BY](#order-by) clause. **Syntax** ```sql SQL theme={null} OFFSET offset_number ``` **Parameters** | **Parameter** | **Description** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | A valid SELECT query. | | `offset_number` | The number of rows to skip when returning results from the query, specified as a positive integer or any constant scalar expression that evaluates to a positive integer. | **Examples** **Skip Rows From the Result Set Using a Number** In this example, the `ORDER BY` statement orders the results chronologically. As a result, the `OFFSET` clause removes the oldest six movies from the result set. ```sql SQL theme={null} SELECT release_date, title FROM movies ORDER BY release_date OFFSET 6; ``` *Output* | release\_date | title | | ------------- | ----------------------- | | 2003-07-09 | Pirates of Palm Springs | | 2003-10-10 | Billy the Killy | | 2003-12-01 | Swords & Scabbards | | 2009-12-10 | CGI Why | | 2010-06-16 | Merchandise Vehicle 5 | | 2010-06-30 | The Last Airman | | 2012-04-25 | The Tentpole | | 2012-07-16 | Superhero 23 | | 2012-10-25 | Spyman 16 | | 2013-04-18 | Iron Chef 3 | | 2013-11-27 | Frigid | | 2015-04-01 | Fury 7 | | 2015-06-09 | Triassic World | | 2015-06-17 | Minnows | **Skip Rows From the Result Set Using an Expression** The `OFFSET` SQL statement can also accept expressions. This query uses the expression `3+4` with the `sys.dummy` table to create a column of incrementing integers by skipping the first seven out of 10 rows. For details, see [Generate Tables Using sys.dummy](/generate-tables-using-sys-dummy). ```sql SQL theme={null} SELECT c1 FROM sys.dummy10 OFFSET 3+4; ``` *Output* ```sql SQL theme={null} 8 9 10 ``` ### INTERSECT Returns any rows that match between two separate `SELECT` queries. To use `INTERSECT`, the two queries must be compatible, meaning they must return the same number of columns and have similar data types. By default, the SQL statement eliminates duplicate rows unless the query includes the optional `ALL` keyword. Using the `DISTINCT` keyword is the same as this default behavior. **Syntax** ```sql SQL theme={null} INTERSECT [ ALL | DISTINCT ] ``` **Parameters** | **Parameter** | **Description** | | ----------------- | --------------------- | | `` | A valid SELECT query. | | `` | A valid SELECT query. | **Example** This example has two `SELECT` queries with an `INTERSECT` statement. The full query retrieves movies that had a budget of at least $200 million, but also earned more than $1 billion in revenues. ```sql SQL theme={null} SELECT * FROM movies WHERE revenue > 1000000000 INTERSECT SELECT * FROM movies WHERE budget > 20000000; ``` *Output* | movie\_id | title | budget | popularity | release\_date | revenue | runtime | movie\_status | vote\_average | vote\_count | | --------- | --------------------- | --------- | ---------- | ------------- | ---------- | ------- | ------------- | ------------- | ----------- | | 597 | Titania | 200000000 | 100.025899 | 1997-11-18 | 1845034188 | 194 | Released | 7.5 | 7562 | | 19995 | CGI Why | 237000000 | 150.437577 | 2009-12-10 | 2787965087 | 162 | Released | 7.2 | 11800 | | 49026 | Superhero 23 | 250000000 | 112.31295 | 2012-07-16 | 1084939099 | 165 | Released | 7.6 | 9106 | | 10193 | Merchandise Vehicle 5 | 200000000 | 59.995418 | 2010-06-16 | 1066969703 | 103 | Released | 7.6 | 4597 | | 211672 | Minnows | 74000000 | 875.581305 | 2015-06-17 | 1156730962 | 91 | Released | 6.4 | 4571 | | 122 | Swords & Scabbards | 94000000 | 123.630332 | 2003-12-01 | 1118888979 | 201 | Released | 8.1 | 8064 | | 37724 | Spyman 16 | 200000000 | 93.004993 | 2012-10-25 | 1108561013 | 143 | Released | 6.9 | 7604 | | 109445 | Frigid | 150000000 | 165.125366 | 2013-11-27 | 1274219009 | 102 | Released | 7.3 | 5295 | | 168259 | Furious 7 | 190000000 | 102.322217 | 2015-04-01 | 1506249360 | 137 | Released | 7.3 | 4176 | | 68721 | Iron Chef 3 | 200000000 | 77.68208 | 2013-04-18 | 1215439994 | 130 | Released | 6.8 | 8806 | | 135397 | Triassic World | 150000000 | 418.708552 | 2015-06-09 | 1513528810 | 124 | Released | 6.5 | 8662 | | 24428 | The Tentpole | 220000000 | 144.448633 | 2012-04-25 | 1519557910 | 143 | Released | 7.4 | 11776 | ### EXCEPT The `EXCEPT` keyword returns the result set of a first `SELECT` query minus any matching rows from a second `SELECT` query. `EXCEPT` requires the two queries to be compatible, meaning they must return the same number of columns and have similar data types. By default, the result set eliminates duplicate rows unless the query includes the optional `ALL` keyword. Using the `DISTINCT` keyword is the same as this default behavior. The `EXCEPT` keyword can also be used to exclude specific columns from a `SELECT *` query. For information on that alternate usage, see the [SELECT](#select) syntax and example. **Syntax** ```sql SQL theme={null} EXCEPT [ ALL | DISTINCT ] ``` **Parameters** | **Parameter** | **Description** | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | A valid SELECT query that is compared to `select_query2`.
The results of an EXCEPT statement are the rows in `select_query1` that do not match any row in `select_query2`. | | `` | A valid SELECT query that is compared to `select_query1`. | **Example** In this example, the first `SELECT` statement queries all rows in the `movies` table. The `EXCEPT` clause and the second `SELECT` statement eliminate from the results any rows with movies that had a budget greater than `20000000`. ```sql SQL theme={null} SELECT * FROM movies EXCEPT SELECT * FROM movies WHERE budget > 20000000; ``` *Output* | title | budget | popularity | release\_date | revenue | runtime | movie\_status | vote\_average | vote\_count | | ---------------------------- | -------- | ---------- | ------------- | --------- | ------- | ------------- | ------------- | ----------- | | Bang Bang Western | 14000000 | 37.380435 | 1992-08-07 | 159157447 | 131 | Released | 7.7 | 1113 | | Space Odyssey 6000 | 10500000 | 86.201184 | 1968-04-10 | 68700000 | 149 | Released | 7.9 | 2998 | | Raiders of the Jungle Temple | 18000000 | 68.159596 | 1981-06-12 | 389925971 | 115 | Released | 7.7 | 3854 | ### UNION Returns the combined result set of two or more `SELECT` queries. By default, `UNION` eliminates duplicate rows from the result set unless you specify the `ALL` keyword. Using the `DISTINCT` keyword is the same as this default behavior. **Syntax** ```sql SQL theme={null} UNION [ ALL | DISTINCT ] [ ... ] ``` **Parameters** | **Parameter** | **Data Type** | **Description** | | --------------- | ------------- | ------------------------------------------------------------- | | `select_query1` | String | A valid `SELECT` query that is combined with `select_query2`. | | `select_query2` | String | A valid `SELECT` query that is combined with `select_query1`. | **Example** This example uses `UNION` to merge two separate queries for identical columns into the same result set. ```sql SQL theme={null} SELECT title, popularity, vote_average FROM movie WHERE popularity > 800 UNION SELECT title, popularity, vote_average FROM movies.movie WHERE vote_average > 7.5; ``` *Output* | title | popularity | vote\_average | | ---------------------------- | ---------- | ------------- | | Raiders of the Jungle Temple | 68.159596 | 7.7 | | Minnows | 875.581305 | 6.4 | | Apocalypse None | 49.973462 | 8 | | Superhero 23 | 112.31295 | 7.6 | | Billy the Killy | 79.754966 | 7.7 | | Bang Bang Western | 37.380435 | 7.7 | | Blade Walker | 94.056131 | 7.9 | | Swords & Scabbards | 123.630332 | 8.1 | | Merchandise Vehicle 5 | 59.995418 | 7.6 | | Space Odyssey 6000 | 86.201184 | 7.9 | ### USING Overrides various system configurations for processing a specified query. The `USING` keyword is required for only the first query override, not for subsequent ones. For descriptions of the supported query configurations, see the parameter table below. **Syntax** ```sql SQL theme={null} USING [ SCHEDULING_PRIORITY = priority_value ] [ MAX_ROWS_RETURNED = max_rows ] [ MAX_ELAPSED_TIME = max_elapsed_time ] [ MAX_TEMP_DISK_USAGE = max_temp_disk_usage ] [ CACHE_MAX_TIME = cache_max_time ] [ CACHE_MAX_BYTES = cache_max_bytes ] [ SERVICE CLASS service_class ] ``` **Parameters** | **Parameter** | **Description** | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | A valid `SELECT` query. | | `priority_value` | An optional clause that specifies the priority at which the query should run.
The priority value is a floating-point number, which indicates the priority compared to the other queries. The priority value does not specify any percentage of resources that the query takes or any specific degree of difference between queries.
If unspecified, defaults to `SCHEDULING_PRIORITY = 1.0`. | | `max_rows` | An optional clause that limits the number of rows returned by a query. If a query exceeds this number of rows, the system kills it.
For example, `max_rows_returned = 10` kills any queries returning more than ten rows. | | `max_elapsed_time` | An optional clause that limits how long a query can run. If a query exceeds this time limit in seconds, the system kills it.
For example, `max_elapsed_time = 10` kills any queries taking longer than 10 seconds. | | `max_temp_disk_usage` | An optional clause that limits the percentage of temporary disk used by a query. If a query exceeds this temporary disk limit, the system kills it.
For example, `max_temp_disk_usage = 10` kills any queries taking more than 10% of temporary disk space. | | `cache_max_time` | An optional clause that affects whether a specified query uses results from the cache rather than executing the query. If there is a cached result with the same query text, executed less than `cache_max_time` seconds ago, the database uses that cached result. The default value is 0, and thus by default, the database does not return any cached results.
The cache considers all SQL Nodes, and if a potential cached result is only available on a different SQL Node, the database redirects the query to that node. If you specify the force attribute on the connection, which disables load balancing and redirection, the database considers only cached results on the current node. | | `cache_max_bytes` | An optional clause that controls whether the database stores the results of any specified query in the cache. The system stores the results of all queries executed using a service class specifying this attribute in the cache if the result size is smaller than this value. You can determine the size of a result set in bytes by querying the `bytes_returned` field of the `completed_queries` virtual table. The default value is 0, and thus by default, the database does not cache any results.
The database caches results in memory on the SQL Nodes. These results are not cached if there is insufficient memory available. | | `service_class` | An optional clause at the end of a query that sets a specific service class to run that query.
For details, see [CREATE SERVICE CLASS](/users-groups-and-service-classes#create-service-class). | **Example** ```sql SQL theme={null} SELECT * FROM sys.dummy10 USING SCHEDULING_PRIORITY = 1.0 MAX_ROWS_RETURNED = 10 MAX_ELAPSED_TIME = 10 MAX_TEMP_DISK_USAGE = 10; ``` *Output* ```sql SQL theme={null} c1 ----------- 1 2 3 4 5 6 7 8 9 10 ``` ### TRACE An optional clause that executes the query, but discards the original result set. Instead, `TRACE` returns a result set of tracing data that describes the execution of the query. Append the `TRACE` SQL statement to the end of a query. The statement can include optional parameters to control its frequency and level of detail. To understand the results of this statement, see [TRACE Results](#trace-results). Trace queries have the same effect to the workload as a query run without the clause. Contact Ocient Support for guidance in using the TRACE SQL statement. **Syntax** ```sql SQL theme={null} TRACE [ FREQUENCY frequency_int ] [ RESOLUTION resolution_int ] ``` **Parameters** | **Parameter** | **Description** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | A valid `SELECT` query. | | `frequency_int` | Controls how often the database samples trace events during query execution in milliseconds.
If unspecified, defaults to 500 milliseconds. | | `resolution_int` | Determines the level of detail captured for a trace or event.
If the resolution of a trace is greater than the resolution of an event, the database records the event.
If unspecified, defaults to 100 milliseconds. | #### TRACE Results Each row of the trace output provides some information about what an operator instance has done in the time between the previous trace sample and the most current sample. | **Column** | **Description** | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `plan_parent_id` | The UUID of the parent of the operator instance in the query plan. If an operator has multiple parents, this field specifies the UUID of one of the parents. | | `plan_op_id` | The UUID of the operator in the query plan. | | `operator_type` | The type of operator that the database uses to sample trace events. | | `node_id` | The UUID of the node where this operator executes. | | `code_id` | The identifier of the VM core where this operator instance executes. Core identifiers are unique for each node. | | `op_id` | The identifier of the operator instance. Identifiers are unique for each node. | | `time` | The time in milliseconds when the database collects the information for the trace event relative to the start of the query. | | `group` | Group related trace events together. | | `sample` | The sample of trace event data. For possible values, see the following table. | | `value` | The value of trace event data. | **Sample and Value Columns** The `sample` and `value` columns describe what occurred in a specified trace period. | **Sample** | **Description** | | --------------------- | ------------------------------------------------------------------------------- | | `ROWS_IN` | The number of rows an operator instance has read from its children operators. | | `ROWS_OUT` | The number of rows the operator instance returns. | | `BLOOM_FILTERED_ROWS` | The number of rows that this operator instance discards by using Bloom filters. | | `SCHEDULE_CYCLE` | The number of cycles in the schedule for this operator instance. | | `OOM_CYCLE` | The number of out-of-memory cycles in the schedule for this operator. | | `INITIALIZE` | The time when the database initializes the operator. | | `FINALIZE` | The time when the database finalizes the operator. | **Example** ```sql SQL theme={null} SELECT * FROM sys.dummy10 TRACE; ``` *Output* | plan\_parent\_id | plan\_op\_id | operator\_type | node\_id | core\_id | op\_id | time | group | sample | value | | ------------------------------------ | ------------------------------------ | ---------------- | ------------------------------------ | -------- | ------ | ---- | ------------------------------------------ | --------------------- | ----- | | 77041d2b-f7ca-49d5-a3f5-4b75b0782ed1 | 498690cf-4a9b-4c1f-affa-417f31fddc2d | RENAME\_OPERATOR | 08ca7b05-1e4d-455f-9f46-2174ed048d33 | 4 | 8441 | 0 | xg::db::vm::operators::operatorTraceEvents | ROWS\_IN | 1 | | 77041d2b-f7ca-49d5-a3f5-4b75b0782ed1 | 498690cf-4a9b-4c1f-affa-417f31fddc2d | RENAME\_OPERATOR | 08ca7b05-1e4d-455f-9f46-2174ed048d33 | 4 | 8441 | 0 | xg::db::vm::operators::operatorTraceEvents | ROWS\_OUT | 1 | | 77041d2b-f7ca-49d5-a3f5-4b75b0782ed1 | 498690cf-4a9b-4c1f-affa-417f31fddc2d | RENAME\_OPERATOR | 08ca7b05-1e4d-455f-9f46-2174ed048d33 | 4 | 8441 | 0 | xg::db::vm::operators::operatorTraceEvents | BLOOM\_FILTERED\_ROWS | 0 | ### TAG An optional clause that adds one or more tags to the SQL query. You can also add tags to a common table expression in the `WITH` clause. After you define tags, you can find them in the `sys.queries` and `sys.completed_queries` system catalog tables. **Syntax** ```sql SQL theme={null} [ ... ] ::= TAG ``` **Parameters** | **Parameter** | **Description** | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `` | A valid `SELECT` query. | | `tag_identifier` | Identifier of the tag for the query. Enclose the identifier in double quotes if it contains any special characters, such as spaces. | **Examples** **SQL Query with One Tag** Select the count of a table with one row and tag the query with the name `count1`. ```sql SQL theme={null} SELECT COUNT(*) FROM sys.dummy1 TAG count1; ``` **SQL Query with Multiple Tags** Select the device model and tag this query with two names `device_model` and `phones`. ```sql SQL theme={null} SELECT device_model FROM system.adtech_flat LIMIT 1 TAG device_model TAG phones; ``` **Common Table Expression with One Tag** Define a common table expression with the `average_budget` tag. You can also add another tag for the whole query `overall_budget`. ```sql SQL theme={null} WITH avg_budget_table (average_budget) AS ( SELECT AVG(budget) FROM movies TAG average_budget ) SELECT title, budget, revenue FROM movies, avg_budget_table WHERE budget > avg_budget_table.average_budget TAG overall_budget; ``` ### String Literals and Escape Sequences All string literals in SQL statements must be enclosed in single quotes. To use a single quote within a string, you can use another single quote as an escape, `''`\*. \* For other escape sequences, include an `e` character before the string literal, i.e., before the opening single quote. This directs the system to recognize escape sequences in the string literal. This means: * All single `\` characters in the string now escape themselves. * Any subsequent character after the `\` is also escaped if it matches an escape sequence (see the table). If you use escape sequences, be prepared that you might need to alter strings that include `\`, such as directory paths. **Supported Escape Sequences** | **Escape Sequence** | **Description** | | ------------------- | -------------------------------------- | | `''` | Single quotation mark | | `\"` | Double quotation mark | | `\n` | Newline character | | `\r` | Carriage return character | | `\f` | Form feed character (i.e., page break) | | `\b` | Backspace | | `\\` | Backslash | | `\t` | Tab | **Examples** These examples show how the system interprets strings with and without escape sequences. **String Without an Escape Sequence** This example selects the simple string literal `'\my\directory\path'` that does not use escape sequences. ```sql SQL theme={null} SELECT '\my\directory\path'; ``` \*Output: \*`\my\directory\path` **String With an Escape Sequence** This example selects the simple string literal `'\my\directory\path'` and uses an escape sequence. The result omits the backslashes. ```sql SQL theme={null} SELECT e'\my\directory\path'; ``` \*Output: \*`mydirectorypath` **String With an Escape Sequence to Retain the Backslashes** This example uses the `'\\my\\directory\\path'` string with an escape sequence to include backslashes in the result. ```sql SQL theme={null} SELECT e'\\my\\directory\\path'; ``` \*Output: \*`\my\directory\path` **Escape Sequences with Regular Expressions** Be careful using escape sequences with strings that also use regular expressions. Escape sequences in SQL syntax override those in regular expressions. In this example, the [REGEXP\_SUBSTR](/character-and-binary-functions#regexp_substr) function uses the regular expression `\w+` to match any word characters. ```sql SQL theme={null} SELECT REGEXP_SUBSTR( 'abcdefghijklmnopqrstuvwxyz', '\w+' ); ``` \*Output: \*`abcdefghijklmnopqrstuvwxyz` The function behaves differently if the regular expression is an escape sequence because it overrides the `\` character. The regular expression searches only for the character `w`. ```sql SQL theme={null} SELECT REGEXP_SUBSTR( 'abcdefghijklmnopqrstuvwxyz', e'\w+' ); ``` \*Output: \*`w` ## Related Links [SQL Reference](/sql-reference) [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference) [Data Definition Language (DDL) Statement Reference](/data-definition-language-ddl-statement-reference) # Data Types Source: https://docs.ocient.com/data-types Reference of Ocient SQL data types, including numeric, character, date and time, boolean, network, geospatial, array, tuple, and matrix data types. The System supports a variety of data types for SQL functions and querying. For information on supported data types, see [Data Types for SQL Functions](#data-types-for-sql-functions). Data type requirements for a specific function can be found in [SQL Reference](/sql-reference) section. SQL statements, such as DDL and DCL commands, use generic data types numeric and string. For details, see [Data Types for SQL Statements](#data-types-for-sql-statements). ## Data Types for SQL Functions The following data types are supported in table columns and queries. This table provides more information on the data types supported by the Ocient System. The example value column gives SQL input examples for the respective data types. The data types in this table apply to SQL functions. The default column values for the Tuple, Matrix, and Array values in a `CREATE TABLE` statement require an alternate syntax than the examples shown in this table. For examples of how to specify these default values, see [Create a Table with All Data Types](/create-table-sql-statement-examples#create-a-table-with-all-data-types). | **Name** | **Description** | **System Limits** | **Example Value** | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | ARRAY | An array of the specified type in the format `TYPE[]`.
The `TYPE` type can be any supported data type besides ARRAY. For example, `INT[]` creates an array of integers. | Maximum memory allocation: 512MiB | `INT[]\(1,2,3)`
`[]\()` can be used for empty arrays. For example: `BIGINT[]\()` | | BIGINT | 8-byte signed integer | Minimum value: `-9223372036854775808`

Maximum value:
`9223372036854775807` | `9876543210` | | BINARY(N) or HASH(N) | Fixed-length binary array with length N | Maximum memory allocation: 512MiB

For HASH(N), the maximum value is N repeats of `0xFF`, and the minimum value is N repeats of `0x00`. | BINARY(4): `'0x01234567'`,
BINARY(3): `'0xabcdef'` | | BOOLEAN | 1-byte logical Boolean value | None | `TRUE`, `FALSE` | | CHARACTER(N) or CHAR(N) or CHAR or CHARACTER | Variable-length character string. Length N is only for compatibility. The Ocient System does not use this argument. | Maximum memory allocation: 512MiB | CHAR(16): `'This is a string'` | | DATE | 4-byte calendar date (year, month, day) | Minimum value: `0001-01-01`

Maximum value:
`9999-12-31` | `'2020-02-02'`,
`'2000-01-01'` | | DECIMAL(P,S) | Exact numerical with precision P and scale S | Minimum value: `-9999999999999999999999999999999`

Maximum value: `9999999999999999999999999999999` | `123.45` | | DOUBLE | 8-byte double precision floating-point number | Minimum value:
-1.7977 x 10^308

Maximum value:
1.7977 x 10^308

| `3.141592` | | FUNCTION | Lambda function or user-defined function | None | `(x int, y int) -> CASE WHEN x = y THEN 0 WHEN INT(COALESCE(x, 1000)) < INT(COALESCE(y, 1000)) THEN -1 ELSE 1 END` | | INT | 4-byte signed integer | Minimum value: `-2147483648`

Maximum value:
`2147483647` | `123456789` | | IPV4 | 4-byte Internet Protocol version 4 | Minimum value:
`0.0.0.0`

Maximum value:
`255.255.255.255` | `'127.0.0.1'` | | IP | 16-byte Internet Protocol version 6 (can also hold IPV4) | Minimum value:
`0000:0000:0000:0000:0000:0000:0000:0000`

Maximum value:
`ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff` | `'0123:4567:89ab:cdef:0123:4567:89ab:cdef'`,
`'1111::0'`,
`'127.0.0.1'` | | LINESTRING
| Geometric type composed of N points | Maximum memory allocation: 512MiB | `'LINESTRING(0 0,2 0)'` | | MATRIX | A one or two-dimensional mathematical matrix used for matrix calculations. | Minimum value: None

Maximum value: 16,384 elements | A Matrix\[2]\[2]:
`{ {0, 0}, {0, 0} }`
A Matrix \[2]\[4]:
`{ {0, 0, 0, 0}, {0, 0, 0, 0} }` | | POINT | Geometric point in 2 dimensions | None | `'POINT(-87.6410 41.8841)'` | | POLYGON | Geometric type composed of N closed linestrings | Maximum memory allocation: 512MiB | `'POLYGON((0 0,2 0,5 5,0 2,0 0), (0 0,1 0,2 2,0 2,0 0))'` | | FLOAT | 4-byte IEEE single precision floating-point number | Minimum value:
-3.4028 x 10^38

Maximum value:
3.4028 x 10^38 | `2.718` | | SMALLINT | 2-byte signed integer | Minimum value: `-32768`

Maximum value:
`32767` | `32767` | | TIME | 8-byte time of day in nanoseconds | Minimum value: `00:00:00.000000000`

Maximum value:
`23:59:59.999999999` | `'12:34:56.012345678'` | | TIMESTAMP | 8-byte date and time in nanoseconds with no associated time zone. | Minimum value: `1677-09-21 00:12:43.145224192`

Maximum value:
`2262-04-11 23:47:16.854775807` | `'2000-01-02 12:34:45'`
| | TINYINT or BYTE | 1-byte signed integer | Minimum value: `-128`

Maximum value:
`127` | `127` | | TUPLE\<\> | Tuple of elements of different types. | Maximum memory allocation: 512MiB | `tuple<>(1,2)`
An Array of Tuples:
`tuple<>[](tuple<>(1,2), tuple<>(3,4))` | | UUID | 16-byte universally unique identifier | Minimum value:
`00000000-0000-0000-0000-000000000000`

Maximum value:
`ffffffff-ffff-ffff-ffff-ffffffffffff` | `'01234567-89ab-cdef-1357-0123456789ab'` | | VARBINARY(N) | Variable-length binary array with maximum length of N | Maximum memory allocation: 512MiB | VARBINARY(6): `'0xaabbccddeeff'` | | VARCHAR(N) | Variable-length character string with maximum length of N | Maximum memory allocation: 512MiB | VARCHAR(4): `'This is a variable length string'` | ## Data Types for SQL Statements DDL, DCL, and General SQL Syntax statements define parameter data types as either numeric or string. These data types are simplified terms that follow these rules: * `numeric` — Any numeric value. This value defaults to the `BIGINT` data type if it is an integer with no decimal parts. If the number includes decimal parts, it defaults to `DOUBLE`. * `string` — A series of one or more characters. This value defaults to the `VARCHAR` data type. ## Data Type Considerations Integer literals in SQL statements are of type BIGINT unless explicitly cast to another type. Floating point literals in SQL statements are of type DOUBLE unless explicitly cast to another type. TRUE and FALSE can be used in SQL statements as valid BOOLEAN literals. Floating point values must be strictly numeric. Infinity and NaN values are unsupported. The decimal type supports a maximum precision of 31 and a scale of one less than the precision. Interval types can only be used within an expression. If the final result of an outermost expression is an interval type, the Ocient System automatically converts it to a BIGINT and the value loses the units information. * "yyyy" : The year as a four-digit number. * "MM" : The month, from 01 through 12. * "dd" : The day of the month, from 01 through 31. * "HH" : The hour, using a 24-hour clock from 00 to 23. * "mm" : The minute, from 00 through 59. * "ss" : The second, from 00 through 59. * "FFFFFFFFF" : If non-zero, the billionths of a second in a date and time value. ## Data Type Compatibility The Ocient System casts data types to the highest common type when possible in expressions to produce an output of that type. These data-type promotions occur in these hierarchies. This promotion often applies to `COALESCE`, `CONCAT`, and `||`, and other operators. **Numeric Types** ```sql SQL theme={null} DOUBLE FLOAT DECIMAL BIGINT INT SMALLINT TINYINT ``` **Geospatial Types** ```sql SQL theme={null} POLYGON LINESTRING POINT ``` Geospatial functions might have semantic differences when you execute these functions with different types that have higher precedence than the type promotion of its arguments. ## Size Limits Variable-length columns have limits to how large the value can be. Data types with a 124 KiB size limit: * TUPLE\<\> : Tuple value with a VARCHAR(N) or VARBINARY(N) type Data types with a 512 MiB size limit: * TYPE\[]: The inner values of the array are subject to the size limits of the inner type. * LINESTRING * POLYGON * VARBINARY(N) * VARCHAR(N) The database ignores length of N for CHAR, VARCHAR, and VARBINARY. The database uses N for SQL conformance only. When you query data, if the database must analyze a value that exceeds 124 KiB at query execution time, the system increases memory usage. Large computed values are non-freeable during query execution, and the system cannot offload to temporary disk. Queries that operate on computed large values with many rows can cause the system to run out of memory and result in the killing of the query. ## Array Array is a SQL type container that stores multiple elements of the same SQL type. Arrays can have zero or any number of values. For instance, an INT array can contain many integers, including NULL. Arrays in Ocient can store any SQL type: IPV4, IP, BOOLEAN, BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, TIME, TIMESTAMP, CHAR, DATE, BINARY, HASH, POINT, UUID, DECIMAL, and TUPLE. Arrays have special functions and operators that are described in the following sections: * [Array Functions](/array-functions-and-operators) * [Array Operators](/array-functions-and-operators#array-operators) ## Tuple Tuple is a SQL type container that is meant to store elements of different types. Ocient supports tuples using any of the supported SQL types: IPV4, IP, BOOLEAN, BYTE, SHORT, INT, BIGINT, FLOAT, DOUBLE, TIME, TIMESTAMP, CHAR, DATE, BINARY, HASH, POINT, UUID, and DECIMAL. In Ocient, tuples can either exist as single columns, SQL array elements, nested tuples, or tuples of arrays. Tuple functions and operators are described in the following sections: * [Tuple Functions](/tuple-functions-and-operators#tuple-functions) * [Tuple Operators](/tuple-functions-and-operators#tuple-operators) ## Matrix Matrix is a SQL-type container. It is a fixed-size, one or two-dimensional array (mathematical matrix) of the same fixed-size SQL data types. Currently, Matrix only supports storing doubles. Matrix dimensions must be non-zero. Matrix functions and operators are described in the following sections: * [Matrix Functions](/matrix-functions-and-operators#matrix-functions) * [Matrix Operators](/matrix-functions-and-operators#matrix-operators) ## Geospatial Data Types Ocient supports three different geospatial geographies: POLYGONS, LINESTRINGS, and POINTS. * A `POLYGON` can be constructed with a closed LINESTRING or an outer shell and an array of inner rings. * A `LINESTRING` represents a series of POINTS connected by line segments. It can be constructed with either LINESTRINGS or POINTS. * A `POINT` represents a point in space defined by an (x, y)/(long, lat) coordinate pair. Polygon semantics in the Ocient System follow the [OGC standard (ISO 19125-1) for LinearRings](https://www.iso.org/standard/40114.html), where the exterior and holes can be meaningfully oriented either counter-clockwise or clockwise. To conceptualize this, imagine walking along the exterior of the polygon in the order of its defined points, so everything to your left is the "interior" of the polygon. A counter-clockwise polygon is the typical orientation where, using an example of a state boundary, the interior of the state corresponds to the interior of the polygon. If you define the state boundary using the clockwise orientation, the interior of the polygon represents everything except the state. If a counter-clockwise exterior polygon has holes, the holes should be clockwise-oriented. Ensure that both the data and polygon literals are oriented in the way you intend when you use them in queries. ### Geography Equivalent Many other geospatial implementations use an object-oriented approach to geospatial data types by implementing a `GEOGRAPHY` union type that might contain any combination of these types: `POINT`, `LINESTRING`, or `POLYGON`. Ocient supports `GEOGRAPHY` slightly differently by using `POLYGON` as an implicit union type. This container type means unclosed `POLYGON` types with no holes are equivalent to `LINESTRING` types, and that a `POLYGON` might contain a single `POINT`. This definition of a `POLYGON` differs from other implementations, where such `POLYGON` types would be considered degenerate. During loading, Ocient is able to convert incoming `GEOGRAPHY` data to the appropriate `POLYGON` equivalent. Ocient geospatial functions appropriately handle these cases as needed. ### Spatial Reference Identifier (SRID) Similar to geography union types, many other geospatial implementations define an attribute table of column identifiers and an associated SRID. The system uses the SRID lookup to define semantics of functions such as `st_area`, which uses different calculations if the specified polygon is planar, defined on an ideal sphere, or on a spheroid. The identifiers also specify units of measure and the number of variables used to define a point in the reference system. However, Ocient only supports the storage of latitude and longitude coordinates for its geography types, and uses only GCS WGS 84 (EPSG code 4326) semantics. Rather than applying a transformation like you might in other implementations to change units or model accuracy, the geospatial measurement functions in the Ocient System allow you to specify the units of measure directly, as well as whether to use a faster idealized sphere model of the Earth, or a more accurate spheroidal model. Ensure that you set the units and spheroidal model correctly when performing critical measurements. Also, be careful when you export data from other databases to Ocient so that the incoming data is in GCS WGS 84 format. Ensure that you use the Loading and Transformation pipeline to prepend or strip SRID definitions from EWKT or EWKB-formatted data. Internally, the Ocient System has no concept of SRIDs, however it does support parsing and emitting SRID-prepended data for compatibility with external tools. When the database parses SRID-prepended data, the system ignores the SRID component and parses the data as a GCS WGS 84 defined geography. ### Geospatial Functions Each geospatial geography has a number of functions and operators that can be used in SQL queries to perform analyses. Some functions apply to specific geography types. For more function and operator specifics, see [Geospatial Functions](/geospatial-functions). ## Related Links [Date and Time Functions](/date-and-time-functions) [Character and Binary Functions](/character-and-binary-functions) [Query Ocient](/query-ocient) [CREATE TABLE SQL Statement Examples](/create-table-sql-statement-examples) # Data Types for Data Pipelines Source: https://docs.ocient.com/data-types-for-data-pipelines Reference for how Ocient data types map to source data formats during data pipeline loads, including conversions, NULL handling, and supported coercions. ## Source Fields When you load data using a data pipeline, you reference extracted data using a special syntax named a source field reference. These source field references can behave differently depending on the data format. It is important to understand how to reference source field data and also how assigns the type of data you reference. ### Source Field References You reference source fields in a pipeline using a named reference such as `$my_field` or a numeric field reference such as `$1`. For some data formats like JSON and , Ocient supports nested data access such as `$my_field.sub_field`. Learn more about what source field references are supported for each data format in [Data Formats for Data Pipelines](/data-formats-for-data-pipelines). ### Source Field Data Types The Ocient System treats references to source fields (e.g., `$my_field`) differently depending on the data format `FORMAT` specified in the data pipeline. **Text-Based Data Formats** For text-based formats such as `DELIMITED` and `JSON`, the Ocient System treats source fields as `VARCHAR`. For `JSON` data, the system treats source fields strictly as `VARCHAR` data even if the field value is represented logically in JSON as Boolean, integer, NULL, or double. Learn more about text-based formats in [Load JSON Data](/data-formats-for-data-pipelines#load-json-data) and [Load Delimited and CSV Data](/data-formats-for-data-pipelines#load-delimited-and-csv-data). **Binary-Based Data Format** For the binary-based format `BINARY`, Ocient treats source fields as the `BINARY` data type. Learn more about binary formats in [Load Binary Data](/data-formats-for-data-pipelines#load-binary-data). **Parquet-Based Data Format** For the Parquet-based format `PARQUET`, Ocient encodes information about the data type of the fields during the load. ## Supported Data Types The data pipeline functionality supports all column data types. `ST_POINT` is not supported in a 3-coordinate format. When you load data into a column type, in many cases, Ocient automatically casts the source data to the target column. However, in some cases, you might need to cast data to the target column type explicitly. Learn more about supported casting functions in [Scalar Transformation Functions and Casting](/transform-data-in-data-pipelines#scalar-transformation-functions-and-casting). ## Automatic Conversion and Casting Loading pipelines apply automatic casting to the final values of your transformations and source fields where the data type does not match the target column type. The casting function that the Ocient System applies varies based on the SQL type of the data that results from your transformation and the SQL type of the target column. The system applies casting automatically in cases to resolve conflicts between data types: * Promotion of a type to expand to a wider data type (e.g., `SMALLINT` to `BIGINT`) * Implicit casting from one family of data type to another (e.g., `VARCHAR` to `TIMESTAMP`) * Ocient supports implicit casting of all simple data types to `VARCHAR`. Auto-casting in Parquet does not support the automatic conversion to `VARCHAR` columns. You must explicitly cast data to the `CHAR` data type when you convert Parquet data that is not string data to a `VARCHAR` column or `VARCHAR` function argument. Automatic casting does not apply in some situations where data loss might occur: * Implicit downcasting to a narrower data type (e.g., `BIGINT` to `SMALLINT`) Where automatic casting is not supported, you can explicitly cast data or use built-in transformation functions according to your requirements. See [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) for more details. ### Implicit Casting Support List **VARCHAR Source Data** The Ocient System treats the source field reference from `JSON` and `DELIMITED` formatted data as `VARCHAR` regardless of the logical type in the source. Ocient supports the implicit casting of `VARCHAR` to all data types. **BINARY Source Data** The system treats the source field reference from `BINARY`-formatted data as raw bytes with automatic casting functions tailored to the `BINARY` data type. Ocient supports implicit casting of `BINARY` to: * `TINYINT`, `SMALLINT`, `INT`, `BIGINT` * `FLOAT`, `DOUBLE` * `TIMESTAMP`, `DATE`, `TIME` (as `VARCHAR` data) For `BINARY` or `VARCHAR` to `BOOL`: * You can represent `true` as: `'t', 'true', 'yes', '1', 'y', 'on'` * You can represent `false` as: `'f', 'false', 'no', '0', 'n', 'off'` For `BINARY` or `VARCHAR` to `TIMESTAMP`, `DATE`, and `TIME`, the automatic casting assumes that the character data has the format: * `TIMESTAMP` — `YYYY-MM-DD HH:mm:ss[.SSSSSSSSS]` * `DATE` — `YYYY-MM-DD` * `TIME` — `HH:mm:ss` **Other Implicit Casts or Downcasts** The system also supports some other implicit casts that are not type promotions: * `TIMESTAMP` data implicitly downcasts to `DATE` and `TIME` columns, truncating the unused portion. * ℹ️This option is available in version 25 or later. * The system implicitly casts all simple types to `VARCHAR` columns except: * `HASH` or `BINARY` * `DECIMAL` * `IP` or `IPV4` For geospatial data types, see [Load Geospatial Data in Data Pipelines](/load-geospatial-data-in-data-pipelines). ### Data Type Promotion List The data type promotion list describes how the Ocient System can automatically promote the specified type to a target column type. The system achieves the promotion using standard casting functions (e.g., `BIGINT(SMALLINT)`). See [Implicit Casting Support List](#implicit-casting-support-list) for implicit casting, for example casting to `VARCHAR`. | **Source Data Type** | **Automatic Casting Precedence for Target Column Type** | | -------------------- | --------------------------------------------------------------------------------------------------------------- | | `BYTE` or `TINYINT` | `BYTE` `TINYINT` `SMALLINT` `INT` `BIGINT`
`FLOAT` `DOUBLE` | | `SMALLINT` | `SMALLINT` `INT` `BIGINT`
`FLOAT` `DOUBLE` | | `INT` | `INT` `BIGINT`
`FLOAT` `DOUBLE` | | `BIGINT` | `BIGINT`
`DOUBLE`
`TIMESTAMP` `TIME` | | `FLOAT` | `FLOAT` `DOUBLE` | | `DOUBLE` | `DOUBLE` | | `DECIMAL` | `DECIMAL` | | `BOOL` | `BOOLEAN` | | `UUID` | `UUID` | | `DATE` | `DATE` `TIMESTAMP` | | `TIME` | `TIME` | | `TIMESTAMP` | `TIMESTAMP`
See [Implicit Casting Support List](#implicit-casting-support-list) for supported downcasting. | | `IPV4` | `IPV4` | | `IP` | `IP` | | `BINARY` | `BINARY`
See [Implicit Casting Support List](#implicit-casting-support-list) for supported cross-casting. | | `ARRAY` | `ARRAY` | | `TUPLE` | `TUPLE` | | `VARCHAR` | `VARCHAR`
See [Implicit Casting Support List](#implicit-casting-support-list) for supported cross-casting. | | `ST_POINT` | `ST_POINT`, `ST_LINESTRING`, `ST_POLYGON` | | `ST_LINESTRING` | `ST_LINESTRING`, `ST_POLYGON` | | `ST_POLYGON` | `ST_POLYGON` | * `BIGINT` to `TIME` automatic casting assumes a time value in milliseconds. * `BIGINT` to `TIMESTAMP` automatic casting assumes milliseconds after the epoch. (January 1, 1970 at midnight UTC) * For complex types, the system applies the type precedence for automatic casting to the elements in the type. * Automatic casting of integral types to DOUBLE and FLOAT are potentially lossy due to limits in precision of floating point representations. * ℹ️This option is available in version 25 or later. The system treats `BYTE` and `TINYINT` aliases identically. ## Related Links [Load Data](/load-data) [Data Formats for Data Pipelines](/data-formats-for-data-pipelines) [Data Pipelines Reference](/data-pipelines) [Load Metadata and File-Based Partitioned Data in Data Pipelines](/load-metadata-and-file-based-partitioned-data-in-data-pipelines) # Data Types for User-Defined Data Pipeline Functions Source: https://docs.ocient.com/data-types-for-user-defined-data-pipeline-functions Reference for Java packages and classes used in Ocient user-defined data pipeline functions, including column type enumerations and array and tuple definitions. This description provides the packages that support the functionality for user-defined data pipeline functions. ## Classes ### `com.ocient.streaming.schema.OcientColumnType` The class that enumerates all defined column types in the Ocient System. | **Enum Value** | | -------------------- | | `TYPE_NONE` | | `TYPE_INFERRED` | | `TYPE_DELETED` | | `TYPE_INT` | | `TYPE_BIGINT` | | `TYPE_FLOAT` | | `TYPE_DOUBLE` | | `TYPE_VARCHAR` | | `TYPE_IPV4` | | `TYPE_TIMESTAMP` | | `TYPE_DATE` | | `TYPE_BOOLEAN` | | `TYPE_BINARY` | | `TYPE_SMALLINT` | | `TYPE_BYTE` | | `TYPE_UUID` | | `TYPE_HASH` | | `TYPE_IP` | | `TYPE_ST_POINT` | | `TYPE_TIME` | | `TYPE_DECIMAL` | | `TYPE_ARRAY` | | `TYPE_TUPLE` | | `TYPE_ST_LINESTRING` | | `TYPE_ST_POLYGON` | ### `com.ocient.streaming.schema.OcientColumnTypeArguments` The class that describes arguments of a column type. Not all column types require arguments. | **Modifier And Type** | **Field and Description** | | ----------------------------------------------- | ------------------------------------------------------------------------------------ | | `public static final OcientColumnTypeArguments` | `NONE` — A default instance indicating no arguments are necessary for a column type. | ### `com.ocient.streaming.schema.OcientColumnTypeArguments.Decimal` A subclass of `OcientColumnTypeArguments` that includes information for `TYPE_DECIMAL`. | **Modifier And Type** | **Field and Description** | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `public int` | `precision` — The number of digits in a number. | | `public int` | `scale` — The number of digits to the right of the decimal point in a number. | | `public boolean` | `fullFormat` — True if the serialized format includes one byte for precision and one byte for scale. You should always set this to `true`. | | **Constructor** | | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `public Decimal(int precision, int scale, boolean fullFormat)` Constructs a newly allocated `OcientColumnTypeArguments` for `OcientColumnType.TYPE_DECIMAL`. | ### `com.ocient.streaming.schema.OcientColumnTypeArguments.Hash` A subclass of `OcientColumnTypeArguments` that includes information for `TYPE_HASH`. | **Modifier And Type** | **Field and Description** | | --------------------- | ------------------------------------------------- | | `public int` | `length` — The number of bytes in the hash value. | | **Constructor** | | ---------------------------------------------------------------------------------------------------------------------- | | `public Hash(int length)` — Constructs a newly allocated `OcientColumnTypeArguments` for `OcientColumnType.TYPE_HASH`. | ### `com.ocient.streaming.schema.OcientColumnTypeArguments.Array` A subclass of `OcientColumnTypeArguments` that includes information for `TYPE_ARRAY`. | **Modifier And Type** | **Field and Description** | | ---------------------------------- | ------------------------------------------------------------------- | | `public OcientColumnType` | `elementType` — The array element type. | | `public OcientColumnTypeArguments` | `elementTypeArguments` — Type arguments for the array element type. | | **Constructor** | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `public Array(OcientColumnType elementType)` — Constructs a newly allocated `OcientColumnTypeArguments`for an array of `elementType`. The `elementTypeArguments` field initializes to `OcientColumnTypeArguments.NONE`. | | `public Array(OcientColumnType elementType, OcientColumnTypeArguments elementTypeArguments, int offsetSizeOverride)` — Constructs a newly allocated `OcientColumnTypeArguments`for an array of `elementType`. You should set `offsetOverride` to `0`. | ### `com.ocient.streaming.schema.OcientColumnTypeArguments.TopLevelTuple` A subclass of tuple that includes information for top-level tuples. | **Modifier And Type** | **Field and Description** | | ---------------------------- | ----------------------------------------------------- | | `private List` | `elementOcientColumns` — List of tuple element types. | | **Modifier And Type** | **Method and Description** | | ---------------------------------------- | -------------------------------------------------------------------------------------------------- | | `public List` | `getTupleTypeArguments()` — Returns the list of tuple element type arguments. | | `public List` | `getTupleTypes()` — Returns the list of tuple element types. | | `public List` | `getElementOcientColumns()` — Returns the list of tuple element types paired with their arguments. | | **Constructor** | | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `public TopLevelTuple(List elementOcientColumns)` — Constructs a newly allocated `OcientColumnTypeArguments` for a `OcientColumnType.TYPE_TUPLE`. | ### `com.ocient.streaming.schema.OcientColumn` Represents an Ocient column with information parsed from metadata of the Ocient System. | **Modifier And Type** | **Method and Description** | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `public String` | `name()` — The name of the column. | | `public OcientColumnType` | `type()` — The data type of the column. | | `public OcientColumnTypeArguments` | `typeArguments()` — Some column types, for example `Decimal`, have arguments that are required to fully understand their format. A relevant subclass of `OcientColumnTypeArguments` returns, as appropriate, or `OcientColumnTypeArguments.NONE`. | | `public boolean` | `required()` — Returns `true` if the user is required to provide this column. This is generally the case for non-defaulted, non-nullable columns. | | `public boolean` | `nullable()` — Returns true if this column is configured as nullable, false otherwise. | ### `com.ocient.streaming.data.types.gis.Coordinate` Represents a two-dimensional cartesian coordinate. | **Modifier And Type** | **Field and Description** | | --------------------- | ------------------------------------ | | `public double` | `x` — The x coordinate of the point. | | `public double` | `y` — The y coordinate of the point. | | **Modifier And Type** | **Method and Description** | | --------------------- | ----------------------------------------------- | | `public double` | `x()` — Returns the x coordinate of this point. | | `public double` | `y()` — Returns the y coordinate of this point. | | **Constructor** | | ---------------------------------------------------------------------------------- | | `public Coordinate(double x, double y)` — Constructs a newly allocated Coordinate. | ### `com.ocient.streaming.data.types.gis.STPoint` Represents a two-dimensional cartesian coordinate. | **Modifier And Type** | **Field and Description** | | --------------------- | ---------------------------------------------------- | | `public Coordinate` | `coordinate` — The ordered, x and y coordinate pair. | | **Modifier And Type** | **Method and Description** | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `public static STPoint` | `fromWKT(String text)` — Parses an `STPoint` from the WKT text. | | `public static STPoint` | `fromEWKT(String text)` — Parses an `STPoint` from the EWKT text. | | `public double` | `getX()` — Returns the x coordinate of this point. | | `public double` | `getY()` — Returns the y coordinate of this point. | | `public STLinestring` | `toLinestring()` — Promotes the `STPoint` to an `STLinestring` container. Useful when representing a geometry collection consisting of `STLinestring` and `STPoint` elements. | | `public STPolygon` | `toPolygon()` — Promotes the `STPoint` to an `STPolygon` container. Useful when representing a geometry collection consisting of any geospatial element type. | | **Constructor** | | --------------------------------------------------------------------------------- | | `public STPoint(Coordinate coordinate)` — Constructs a newly allocated `STPoint`. | ### `com.ocient.streaming.data.types.gis.STLinestring` Geometric type that is composed of a sequence of N points. | **Modifier And Type** | **Field and Description** | | --------------------- | ------------------------------------------------------------ | | `public STPoint[]` | `points` — The ordered sequence of points in the linestring. | | **Modifier And Type** | **Method and Description** | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `public static STLinestring` | `fromWKT(String text)` — Parses an `STLinestring` from the WKT text. | | `public static STLinestring` | `fromEWKT(String text)` — Parses an `STLinestring` from the EWKT text. | | `public STPoint[]` | `points()` — Returns the points of this linestring. | | `public STLinestring` | `toLinestring()` — Returns `this`. | | `public STPolygon` | `toPolygon()` — Promotes the `STLinestring` to an `STPolygon` container. Useful when representing a geometry collection consisting of any geospatial element type. | | **Constructor** | | -------------------------------------------------------------------------------------- | | `public STLinestring(STPoint[] points)` — Constructs a newly allocated `STLinestring`. | ### `com.ocient.streaming.data.types.gis.STPolygon` Geometric type composed of N closed linestrings. | **Modifier And Type** | **Field and Description** | | ----------------------- | ---------------------------------------------------- | | `public STLinestring` | `exteriorRing` — The exterior ring of the polygon. | | `public STLinestring[]` | `interiorRings` — The interior rings of the polygon. | | **Modifier And Type** | **Method and Description** | | ---------------------------- | ------------------------------------------------------------------- | | `public static STLinestring` | `fromWKT(String text)` — Parses an `STPolygon` from the WKT text. | | `public static STLinestring` | `fromEWKT(String text)` — Parses an `STPolygon` from the EWKT text. | | `public STLinestring` | `exteriorRing()` — Returns the exterior linestring. | | `public STLinestring[]` | `interiorRings()` — Returns the interior linestrings. | | `public STPolygon` | `toPolygon()` — Returns `this`. | | **Constructor** | | ----------------------------------------------------------------------------------------------------------------------- | | `public STPolygon(STLinestring exteriorRing, STLinestring[] interiorRings)` — Constructs a newly allocated `STPolygon`. | ### `com.ocient.streaming.data.types.Timestamp` Timestamp represented as nanoseconds from the epoch. | **Modifier And Type** | **Method and Description** | | --------------------- | ------------------------------------------------------------------- | | `public long` | `value()` — Returns the number of nanoseconds after the Unix epoch. | | **Constructor** | | ----------------------------------------------------------------------------------- | | `public Timestamp(long nanosFromEpoch)` — Constructs a newly allocated `Timestamp`. | ### `com.ocient.streaming.data.types.Time` Time that is represented as nanoseconds from midnight (`00:00:00`). | **Modifier And Type** | **Method and Description** | | --------------------- | ------------------------------------------------------------- | | `public long` | `value()` — Returns the number of nanoseconds after midnight. | | **Constructor** | | ---------------------------------------------------------------------------- | | `public Time(long nanosFromMidnight)` — Constructs a newly allocated `Time`. | ### `com.ocient.streaming.data.types.Decimal` A class representing a decimal number with binary-coded decimal (BCD) storage, supporting specified precision and scale. | **Modifier And Type** | **Field and Description** | | --------------------- | ------------------------------------------------------------------ | | `public static long` | `MAX_PRECISION` — The maximum supported precision in the database. | | **Modifier And Type** | **Method and Description** | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `public static Decimal` | `fromString(String s, byte precision, byte scale)`
Creates a `Decimal` object from a string representation with the specified precision and scale.
  • `s` — The string representation of the decimal (e.g., `"123.45"`, `"-0.1"`).
  • `precision` — The precision (total number of digits) of the decimal.
  • `scale` — The scale (number of digits after the decimal point) of the decimal.
| | `public static Decimal` | `fromBigDecimal(java.math.BigDecimal d, byte precision, byte scale)`
Creates a `Decimal` object from a `BigDecimal`, ensuring compatibility with the specified precision and scale.
  • `d` — The `BigDecimal` to convert.
  • `precision` — The precision (total number of digits) of the decimal.
  • `scale` — The scale (number of digits after the decimal point) of the decimal.
| | `public static Decimal` | `fromDouble(double d, byte precision, byte scale)`
Creates a `Decimal` object from a double value with the specified precision and scale.
  • `d` — The double to convert.
  • `precision` — The precision (total number of digits) of the decimal.
  • `scale` — The scale (number of digits after the decimal point) of the decimal.
| | `public String` | `toString()` — Returns the `Decimal` string representation of this `Decimal`. | | `public long` | `asLong()` — Converts this `Decimal` object to a long, truncating any fractional part. | | `public double` | `asDouble()` — Converts this `Decimal` object to a double, preserving the full value including the fractional part. | | `public byte[]` | `bcd()` — Returns the BCD representation of this `Decimal` object. | | `public byte[]` | `getFullFormat()` — Returns the full format representation, including precision and scale bytes followed by the BCD data padded to `MAX_PRECISION`. | | `public int` | `precision()` — Returns the precision of this `Decimal` object. | | `public int` | `scale()` — Returns the scale of this `Decimal` object. | ### `com.ocient.streaming.data.types.OcientTuple` A class representing a tuple with a defined schema of column types and associated values. Supports fixed-length and variable-length elements. | **Modifier And Type** | **Method and Description** | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `public int` | `numValues()` — Returns the number of values currently stored in this tuple. | | `public void` | `setValue(int index, Object value)`
Sets or updates the value at the specified index, ensuring type compatibility.
  • `index` — The index of the value to set (0-based).
  • `value` — The value to set must match the column type at the index.
| | `public void` | `set(List values)`
Sets or resets all values in the tuple from the provided list.
  • `values` — The list of values to set must match the schema size and types.
| | `public String` | `toString()` — Returns a string representation of this tuple, including its schema and values (e.g., `"TUPLE(1,2.0)"`). | | `public List` | `get()` — Returns the list of values in this tuple, in order. | | `public List` | `getTypes()` — Returns the list of column types defining the schema of this tuple, in order. | | `public List` | `getTypeArgs()` — Returns the list of type arguments corresponding to the column types of this tuple, in order. | | **Constructor** | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `public OcientTuple(List types, List typeArgs)`
Constructs an empty tuple with the specified schema of column types and type arguments.
  • `types` — The list of column types defining the tuple schema.
  • `typeArgs` — The list of type arguments corresponding to each column type.
| ## Related Links [CREATE OR REPLACE PIPELINE FUNCTION](/data-pipelines#create-pipeline-function) # Database Administration Source: https://docs.ocient.com/database-administration Administer Ocient databases with tasks for user management, permissions, backups, monitoring, performance tuning, and ongoing operational maintenance. Database Administrators are responsible for ensuring the smooth operation of an System and learning how to maximize performance to meet the goals of their organization. See these pages for key concepts that are useful for understanding Ocient System dynamics and managing users. ## Tutorials ### Security These pages explain system authentication, SSO, network security, and encryption options. * [Ocient Security Guide](/ocient-security-guide) * [Set Up Data Encryption](/set-up-data-encryption) * [Authentication Methods](/authentication-methods) * [Database Password Security Settings](/database-password-security-settings) ### User Control These pages explain Ocient user management, including controlling security and priority for user groups: * [Manage Users, Groups, and Roles](/manage-users-groups-and-roles) * [Workload Management and Service Classes](/workload-management-and-service-classes) * [Workload Management Walkthrough](/workload-management-walkthrough) * [Object-Type Level Privileges Management](/object-type-level-privileges-management) ### Query Performance and Tuning Configure keys and indexes before loading large amounts of data into a table. These pages provide detailed information on how to optimize query performance by setting up Ocient keys and indexing: * [TimeKeys and Clustering Keys](/timekeys-and-clustering-keys) * [Secondary Indexes](/secondary-indexes) * [CREATE TABLE SQL Statement Examples](/create-table-sql-statement-examples) * [Manage Distributed Tasks](/manage-distributed-tasks) ### Storage These pages explain compression and caching options in the Ocient System: * [Table Compression Options](/table-compression-options) * [Global Dictionary Compression](/global-dictionary-compression) * [Table Retention Policies](/table-retention-policies) * [Result Set Caching](/result-set-caching) * [Remove Records from an Ocient System](/remove-records-from-an-ocient-system) ## SQL Reference Pages For detailed documentation on supported DDL or DCL commands and syntax, see: * [Data Query Language (DQL) Statement Reference](/data-query-language-dql-statement-reference) * [Data Definition Language (DDL) Statement Reference](/data-definition-language-ddl-statement-reference) * [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference) ## Related Links [Connect to Ocient](/connect-to-ocient) [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # Database Password Security Settings Source: https://docs.ocient.com/database-password-security-settings Configure password security in Ocient with controls for expiration, lockout, complexity, history, and user account states to meet compliance requirements. You can manage password security in the by using a variety of settings and managing user states. ## Password Security Settings includes password security settings to meet specific security requirements or preferences. Five settings are available to configure your password security. | **Security Setting** | **Description** | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `password_minimum_length` | The minimum length of passwords. The maximum password length supported by the Ocient System is 32. | | `password_complexity_level` | An integer value representing the password complexity algorithm.
Supported values are:
`1` — Level 1. The password must contain at least one upper case character, lower case character, and number.
`2` — Level 2. In addition to the requirements specified in level 1, the password must contain at least one non-alphanumeric character. | | `password_no_repeat_count` | The number of unique passwords that a user must use before they can reuse a password. Even if this setting is set to `0`, when the system determines that the password lifetime has been reached, you must change the password to a unique value. | | `password_lifetime_days` | The password must be changed after this number of days. After the password is older than this period, the user changes to the `PASSWORD_EXPIRED` state on their next login. | | `password_invalid_attempt_limit` | The number of login attempts with an invalid password before a user changes to the `DISABLED` state. | Except for the `password_no_repeat_count` setting, a value of 0 for any of these settings means that the system ignores that setting. ### Password Security Setting Hierarchy and Precedence You can set all these settings at the system, database, or group levels. The Ocient System uses the most restrictive value. For example, if `password_minimum_length` is `8` at the system level, `10` at the database level, and `12` at the group level, the system applies the value `12` to the user. You can only add a user to a group after you create the user in the system. Upon user creation, `password_minimum_length`, `password_complexity_level`, and `password_no_repeat_count` settings are based only on system- and database-level settings. ## System Catalog Table for Security Settings The `sys.security_settings` system catalog table shows current security settings. The table contains settings for only databases and groups if any settings are non-zero. After setting the `password_invalid_attempt_limit` value, you can inspect the value using the `sys.security_settings` system catalog table. ```sql SQL theme={null} SELECT password_invalid_attempt_limit FROM sys.security_settings; ``` The `sys.users` system catalog table contains information about users, their current state, and details about their security status, such as the last time the password was updated or the number of failed login attempts. ## Password Recovery The Ocient System does not enable you to recover a password. If you forget your password, contact the user who has the Security Administrator or Database Administrator role. Users with those roles can set a new password. ## User States Ocient local users, not SSO-based users, are in one of these states: * `ENABLED` * `DISABLED` * `PASSWORD_EXPIRED` ### `ENABLED` State Enabled users have normal access privileges to the system. They can connect and execute SQL statements. This state is the default state for all users. ### `DISABLED` State A disabled user cannot connect to the system, and if they are currently connected, they cannot execute any SQL statements. Users become disabled automatically if they exceed the `password_invalid` number of failed password attempts. An administrator must set the state of the user to `ENABLED` for the user to resume access. ### `PASSWORD_EXPIRED` State This state enables a user to log on. However, the only SQL statement they can execute is `ALTER USER SET PASSWORD='yyyy'`. The user receives a warning that their password is expired when they authenticate. Whenever the `password_lifetime_days` number of days has elapsed after the last time the password changed, the system automatically transitions a user to the `PASSWORD_EXPIRED` state on their next login. You can inspect the last password change timestamp in the `password_updated_at` column in the `sys.users` system catalog table. ### Change User State To change the state of the user to these different states, use the [ALTER USER](/users-groups-and-service-classes#alter-user) SQL statement. ## Related Links [Authentication Methods](/authentication-methods) [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference) [ALTER USER](/users-groups-and-service-classes) [Cluster and Node Management](/cluster-and-node-management) [Users, Groups, and Service Classes](/users-groups-and-service-classes) # Databases Source: https://docs.ocient.com/databases Reference for managing databases in Ocient using SQL DDL, including CREATE DATABASE, ALTER DATABASE, and DROP DATABASE syntax with examples and options. This group of DDL SQL statements allows database administrators to manage databases. Database administrators can create and modify databases, including SSO authentication settings. You can view information about databases using the `sys.databases` system catalog table. For information on database components, see the pages on [Schemas](/schemas), [Tables](/tables), [Views](/views), and [Indexes](/indexes). ## CREATE DATABASE `CREATE DATABASE` creates a new database. The database name must be distinct from the name of any existing database in the system. To create a database, you must have the `CREATE DATABASE` privilege for the current system. **Syntax** ```sql SQL theme={null} CREATE DATABASE [ IF NOT EXISTS ] database_name ``` | **Parameter** | **Type** | **Description** | | --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `database_name` | string | A unique identifier for the database. The system generates an error if you specify a duplicate name.
`system` is a reserved database name and can neither be created nor dropped. | **Example** Create a new database named `ocient`. ```sql SQL theme={null} CREATE DATABASE ocient; ``` ## DROP DATABASE `DROP DATABASE` removes an existing database. This SQL statement also disconnects all users currently connected to the database. To remove a database, you must have the `DROP DATABASE` privilege for the current database. You cannot drop a database while it has any `PIPELINE` in a running status. The `DROP DATABASE` SQL statement removes the existing database and all created users, tables, and views. This action cannot be undone. **Syntax** ```sql SQL theme={null} DROP DATABASE [ IF EXISTS ] database_name [, ...] ``` | **Parameter** | **Type** | **Description** | | --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `database_name` | string | An identifier for the database to be dropped.
You can drop multiple databases by specifying additional database names and separating each with commas.
`system` is a reserved database name and can neither be created nor dropped. | **Example** Remove an existing database named `ocient`. ```sql SQL theme={null} DROP DATABASE ocient; ``` ## ALTER DATABASE ### ALTER DATABASE RENAME `ALTER DATABASE RENAME` renames an existing database. To rename a database, you must have the `ALTER DATABASE` privilege for the database. **Syntax** ```sql SQL theme={null} ALTER DATABASE old_database_name RENAME TO new_database_name ``` | **Parameter** | **Type** | **Description** | | ------------------- | -------- | -------------------------------------------------- | | `old_database_name` | string | The old identifier of the database for the rename. | | `new_database_name` | string | The new identifier of the database for the rename. | **Example** Rename an existing database named `oracle` to `ocient`. ```sql SQL theme={null} ALTER DATABASE oracle RENAME TO ocient; ``` ### ALTER DATABASE SET SSO INTEGRATION `ALTER DATABASE SET SSO INTEGRATION` configures the database to authenticate using an external SSO provider. This SSO integration is the default for connections unless you use a connectivity pool or specify a different provider. To set a connection, you must be a system-level user or a database administrator and have an open connection to the database. This SQL statement is an alias for [ALTER DATABASE ALTER SSO INTEGRATION](#alter-database-alter-sso-integration). See [Configuring the Ocient Database](/authentication-methods#sso-parameters) for details about configuring SSO protocols. If your System is version 25.0 or later, you can create multiple SSO integrations for each database. An SSO integration assigned to the database by the `ALTER DATABASE` SQL statement is the primary SSO connection, unless you connect to the database with a connectivity pool that has a different SSO integration assigned to it. **Syntax** ```sql SQL theme={null} ALTER DATABASE database ALTER SSO INTEGRATION sso_name ``` | **Parameter** | **Type** | **Description** | | ------------- | -------- | ------------------------------------------------- | | `database` | string | The identifier of the database for configuration. | | `sso_name` | string | The identifier of the SSO integration to use. | **Example** This example sets an example database to use the SSO integration named `sso_test`. ```sql SQL theme={null} ALTER DATABASE example_database SET SSO INTEGRATION sso_test; ``` ### ALTER DATABASE ALTER SSO INTEGRATION `ALTER DATABASE ALTER SSO CONNECTION` configures the database to authenticate using an external SSO provider. This SSO integration is the default for connections unless you use a connectivity pool or specify a different provider. To alter a connection, you must be a system-level user or a database administrator and have an open connection to the database. This SQL statement is an alias for [ALTER DATABASE SET SSO INTEGRATION](#alter-database-set-sso-integration). See [Configuring the Ocient Database](/authentication-methods) for details about configuring SSO protocols. If your Ocient System is version 25.0 or later, you can create multiple SSO integrations for each database. An SSO integration assigned to the database by the `ALTER DATABASE` SQL statement is the primary SSO connection, unless you connect to the database with a connectivity pool that has a different SSO integration assigned to it. **Syntax** ```sql SQL theme={null} ALTER DATABASE database ALTER SSO INTEGRATION sso_name ``` | **Parameter** | **Type** | **Description** | | ------------- | -------- | ------------------------------------------------- | | `database` | string | The identifier of the database for configuration. | | `sso_name` | string | The identifier of the SSO integration to use. | **Example** This example alters an example database to use the SSO integration named `sso_test`. ```sql SQL theme={null} ALTER DATABASE example_database ALTER SSO INTEGRATION sso_test; ``` ### ALTER DATABASE REMOVE SSO INTEGRATION `ALTER DATABASE REMOVE SSO INTEGRATION` removes an existing SSO integration as the default connection protocol for the database. This action effectively undoes the [ALTER DATABASE ALTER SSO INTEGRATION](#alter-database-alter-sso-integration) SQL statement. To remove a connection, you must be a system-level user or a database administrator. **Syntax** ```sql SQL theme={null} ALTER DATABASE database REMOVE SSO INTEGRATION ``` | **Parameter** | **Type** | **Description** | | ------------- | -------- | -------------------------------------------- | | `database` | string | The identifier of the database for deletion. | **Example** Remove the default connection from the database named `example_database`. ```sql SQL theme={null} ALTER DATABASE example_database REMOVE SSO INTEGRATION; ``` ### ALTER DATABASE ALTER SECURITY Sets the security settings at the database level using the `ALTER DATABASE ALTER SECURITY` SQL statement. Replace `` with the security setting and `` with the value. **Syntax** ```sql SQL theme={null} ALTER DATABASE database ALTER SECURITY [=] ``` | **Parameter** | **Data** **Type** | **Description** | | ------------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `database` | string | The identifier of the database for setting security settings. | | `security_setting` | string | The security setting with values:
\* `password_minimum_length`
\* `password_complexity_level`
\* `password_no_repeat_count`
\* `password_lifetime_days`
\* `password_invalid_attempt_limit`
For details about these values, see [Database Password Security Settings](/database-password-security-settings). | | `value` | numeric | An integer to represent one of the security settings. For details about this value, see [Database Password Security Settings](/database-password-security-settings). | **Example** Set the password lifetime to 20 days for the database `example_db`. ```sql SQL theme={null} ALTER DATABASE example_db ALTER SECURITY password_lifetime_days = 20; ``` ## Related Links [Core Elements of an Ocient System](/core-elements-of-an-ocient-system) [Data Query Language (DQL) Statement Reference](/data-query-language-dql-statement-reference) [Database Password Security Settings](/database-password-security-settings) [System Catalog](/system-catalog) # Date and Time Functions Source: https://docs.ocient.com/date-and-time-functions Reference for Ocient SQL date and time functions, including timestamp arithmetic, formatting, parsing, time zone conversion, and date part extraction. ## Basic Date and Time Functions ### ADD\_MONTHS Adds the specified number of months to the date. Equivalent to using a date and adding a `MONTH(value)`. **Syntax** ```sql SQL theme={null} ADD_MONTHS(time, int) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function modifies by adding a specified number of months. | | `int` | `INT` | The number of months to add to the `time` value. | **Example** ```sql SQL theme={null} SELECT ADD_MONTHS('2022-11-15 04:18:00',2); ``` *Output*: `2023-01-15` ### CENTURY Returns the number of centuries. **Syntax** ```sql SQL theme={null} CENTURY(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------ | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes to return the number of its centuries. | **Example** In this example, the current date is `2023-01-03`. ```sql SQL theme={null} SELECT CENTURY(CURRENT_DATE()); ``` *Output*: `21` ### CURDATE Alias for CURRENT\_DATE. ### CURRENT\_DATE Alias for CURDATE. Returns the current date in the format `YYYY-MM-DD`. **Syntax** ```sql SQL theme={null} CURRENT_DATE ``` **Example** ```sql SQL theme={null} SELECT CURRENT_DATE; ``` *Output*: `2023-01-03` ### CURRENT\_TIME Returns the current time as a TIME value (e.g., `hh:mm:ss.mm`). **Syntax** ```sql SQL theme={null} CURRENT_TIME ``` **Example** ```sql SQL theme={null} SELECT CURRENT_TIME; ``` *Output:* `19:40:04` ### CURRENT\_TIMESTAMP Alias for [NOW](#now). Returns the current date and time as a TIMESTAMP value (e.g., `YYYY-MM-DD hh:mm:ss.mmm`). **Syntax** ```sql SQL theme={null} CURRENT_TIMESTAMP ``` **Example** ```sql SQL theme={null} SELECT CURRENT_TIMESTAMP; ``` *Output*: `2023-01-03 11:00:03.316674397` ### DATE\_PART Alias for [EXTRACT](#extract). ### DATE\_TRUNC Returns the date or timestamp entered, truncated to the specified precision. **Syntax** ```sql SQL theme={null} DATE_TRUNC(precision, time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `precision` | `CHAR` | The timestamp unit used to truncate the returned value.
For the `precision` input, acceptable values are:
`NANOSECOND`
`MICROSECOND`
`MILLISECOND`
`SECOND`
`MINUTE`
`HOUR`
`DAY`
`WEEK`
`MONTH`
`QUARTER`
`YEAR`
`DECADE`
`CENTURY`
`MILLENNIUM` | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function truncates to the specified precision.
The `time` input supports precision values DAY or larger. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-01-03 11:00:59.114058182`. ```sql SQL theme={null} SELECT DATE_TRUNC('NANOSECOND', CURRENT_TIMESTAMP()); ``` *Output*: `2023-01-03 11:01:05.280975906` **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-01-01 11:00:59.114058181`. ```sql SQL theme={null} SELECT DATE_TRUNC('MONTH', CURRENT_TIMESTAMP()); ``` *Output*: `2023-01-01 00:00:00.000000000` **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-01-01 11:00:59.114058181`. ```sql SQL theme={null} SELECT DATE_TRUNC('CENTURY', CURRENT_TIMESTAMP()); ``` *Output*: `2001-01-01 00:00:00.000000000` ### DAY Alias for [DAY\_OF\_MONTH](#day_of_month). ### DAY\_OF\_WEEK Returns an integer, in the range of 1 to 7, that represents the day of the week. The value 1 is Sunday, and 7 is Saturday. **Syntax** ```sql SQL theme={null} DAY_OF_WEEK(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ---------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for its day of the week. | **Example** ```sql SQL theme={null} SELECT DAY_OF_WEEK(CURRENT_DATE()); ``` *Output*: `3` ### DAY\_OF\_YEAR Alias for DOY. Returns an integer in the range 1 to 366 that represents the day of the year. **Syntax** ```sql SQL theme={null} DAY_OF_YEAR(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------ | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for its day of year. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-01-03 11:00:59.114058182`. ```sql SQL theme={null} SELECT DAY_OF_YEAR(CURRENT_TIMESTAMP()); ``` *Output*: `3` ### DAY\_OF\_MONTH Alias for DAY. Extracts the day-of-month portion of a timestamp or date as an integer. **Syntax** ```sql SQL theme={null} DAY_OF_MONTH(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ----------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for its day of the month. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-01-03 11:00:59.114058182`. ```sql SQL theme={null} SELECT DAY(CURRENT_TIMESTAMP()); ``` *Output*: `3` ### DECADE The decade is the year divided by 10. **Syntax** ```sql SQL theme={null} DECADE(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for its decade. | **Example** This example uses `CURRENT_DATE` set to `2023-01-03`. ```sql SQL theme={null} SELECT DECADE(CURRENT_DATE()); ``` *Output*: `202` ### DOW Alias for [DAY\_OF\_WEEK](#day_of_week). ### DOY Alias for [DAY\_OF\_YEAR](#day_of_year). ### EOMONTH Returns the last day of the month using the specified timestamp or date. If you specify both arguments, this function returns the last day of the month for the resulting timestamp or date after the system adds the specified number of months to the first argument value. **Syntax** ```sql SQL theme={null} EOMONTH(time [, int]) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for the last day of the month. | | `int` | `INT` | Optional. An integer value that represents the number of months to add to the `time` value.
If unspecified, the value defaults to 0. | **Examples** This example returns the last day of the month for the current timestamp. ```sql SQL theme={null} SELECT EOMONTH(CURRENT_TIMESTAMP(),0); ``` *Output*: `2023-02-27` This example returns the last day of the month in November in 2022 by using the EOMONTH function to add one month to the last day of October in 2022. ```sql SQL theme={null} SELECT EOMONTH(DATE('2022-10-31'), 1); ``` *Output*: `2022-11-30` ### EPOCH The number of seconds after 1970-01-01 00:00:00 UTC. **Syntax** ```sql SQL theme={null} EPOCH(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for the number of seconds after 1970-01-01 00:00:00 UTC. | **Example** ```sql SQL theme={null} SELECT EPOCH(CURRENT_TIMESTAMP()); ``` *Output*: `1672743901` ### EXTRACT Extract a component from a timestamp or date. **Syntax** ```sql SQL theme={null} EXTRACT(precision FROM time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for the component to be extracted. | | `precision` | `CHAR` | The timestamp unit to be extracted.
For the `precision` input, acceptable values are:
`NANOSECOND`
`MICROSECOND`
`MILLISECOND`
`SECOND`
`MINUTE`
`HOUR`
`DAY`
`WEEK`
`MONTH`
`QUARTER`
`YEAR`
`DECADE`
`CENTURY`
`MILLENNIUM` | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-01-03 11:00:59.114058182`. ```sql SQL theme={null} SELECT EXTRACT(HOUR FROM CURRENT_TIMESTAMP()); ``` *Output*: `11.0` ### HOUR Extracts the hour portion of a timestamp as an integer. **Syntax** ```sql SQL theme={null} HOUR(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ----------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for its hour value. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-01-03 11:00:59.114058182`. ```sql SQL theme={null} SELECT HOUR(CURRENT_TIMESTAMP()); ``` *Output*: `11` ### ISODOW Extracts the day of the week based on ISO 8601, which ranges from Monday (1) to Sunday (7). **Syntax** ```sql SQL theme={null} ISODOW(timestamp or date) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------------ | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for its day-of-week value. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-01-03 11:00:59.114058182`. ```sql SQL theme={null} SELECT ISODOW(CURRENT_TIMESTAMP()); ``` *Output*: `2` ### ISDATE Returns `TRUE` if the input argument can be successfully cast to a date, `FALSE` otherwise. **Syntax** ```sql SQL theme={null} ISDATE(char) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ----------------------------- | | `char` | `CHAR` | A string to cast as a `DATE`. | **Example** ```sql SQL theme={null} SELECT ISDATE('2023-01-03'); ``` *Output*: `TRUE` ### MAKEDATETIME Returns a timestamp consisting of the specified date and time. **Syntax** ```sql SQL theme={null} MAKEDATETIME(date, time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------- | | `date` | `CHAR` | A date string for timestamp creation. | | `time` | `CHAR` | A time string for timestamp creation. | **Example** ```sql SQL theme={null} SELECT MAKEDATETIME('2022-11-02','01:24:58'); ``` *Output*: `2022-11-02 01:24:58.000000000` ### MILLISECOND Extracts the millisecond portion of a timestamp as an integer. **Syntax** ```sql SQL theme={null} MILLISECOND(timestring) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------ | | `timestring` | `TIMESTAMP` | A time value, specified as a timestamp or date, which the function extracts for the millisecond value. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-02-22 13:59:25.286`. ```sql SQL theme={null} SELECT MILLISECOND(CURRENT_TIMESTAMP()); ``` *Output*: `286` ### MINUTE Extracts the minute portion of a timestamp or date as an integer. **Syntax** ```sql SQL theme={null} MINUTE(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function extracts for the minute value. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-02-22 14:00:40.764`. ```sql SQL theme={null} SELECT MINUTE(CURRENT_TIMESTAMP()); ``` *Output*: `0` ### MONTH\_NAME Returns the calendar name in English of the month for the specified date. Syntax ```sql SQL theme={null} MONTH_NAME(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes its month name. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-02-22 13:59:25.286`. ```sql SQL theme={null} SELECT MONTH_NAME(CURRENT_TIMESTAMP()); ``` *Output*: `February` ### MONTHS\_BETWEEN Returns the difference between the two dates or timestamps in months as a `DOUBLE`. The fractional months component is based on a 31-day month. If the two dates have the same day or are both the last day of the month, this function returns a whole number instead. **Syntax** ```sql SQL theme={null} MONTHS_BETWEEN(time1, time2) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | -------------------------------------------------------------------------------------------------------------------- | | `time1` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function compares to `time2` for the difference in months. | | `time2` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function compares to `time1` for the difference in months. | **Example** ```sql SQL theme={null} SELECT MONTHS_BETWEEN(CURRENT_TIMESTAMP(), ADD_MONTHS(CURRENT_TIMESTAMP(),11)); ``` *Output*: `-11` ### MONTH Extracts the month portion of a timestamp or date as an integer. **Syntax** ```sql SQL theme={null} MONTH(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------ | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes for its month value. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-02-22 13:59:25.286`. ```sql SQL theme={null} SELECT MONTH(CURRENT_TIMESTAMP()); ``` *Output*: `2` ### MSECS The seconds field, including fractional parts. This is the same as `EXTRACT(MILLISECONDS FROM time)`. **Syntax** ```sql SQL theme={null} MSECS(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | --------------------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date. The function multiplies the seconds part of the value by 1,000. | **Example** ```sql SQL theme={null} SELECT MSECS(CURRENT_TIMESTAMP()); ``` *Output*: `37637.648` ### NANOS\_TO\_TIMESTAMP Convert a number of nanoseconds into a timestamp equivalent to the duration after the epoch time, 1970-01-01 00:00:00 UTC. **Syntax** ```sql SQL theme={null} NANOS_TO_TIMESTAMP(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `ns` | `INT` | A number of nanoseconds, specified as an integer, which the function converts into a timestamp equivalent to the duration after the epoch time. | **Example** ```sql SQL theme={null} SELECT NANOS_TO_TIMESTAMP(10000000); ``` *Output*: `1969-12-31 16:00:00.010` ### NEXT\_DAY Returns the closest date after a specified date that lies on a specific day of the week. **Syntax** ```sql SQL theme={null} NEXT_DAY(date, character) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `date` | `DATE`, `TIMESTAMP`, or `CHAR` | A time value, specified as a date, timestamp, or string, which the function uses to return the closest date for a specific day of the week. Can be any type that can be cast to `DATE`. | | `character` | `CHAR` | A string representing a day of the week.
The string must match the first three characters (case insensitive) of the English name of any day of the week. If this string does not match the prefix of any day, this function returns NULL. | **Example** ```sql SQL theme={null} SELECT NEXT_DAY('2023-02-27', 'sat'); ``` *Output*: `2023-03-04` ### NOW Alias for [CURRENT\_TIMESTAMP](#current_timestamp). ### QUARTER Returns an integer between 1 and 4 representing the quarter of the year in which the specified date falls. **Syntax** ```sql SQL theme={null} QUARTER(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | -------------------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function computes to return the quarter of the year. | **Example** ```sql SQL theme={null} SELECT QUARTER(DATE('2023-02-08 09:00:00.000')); ``` *Output*: `1` ### ROUND Returns the specified date or timestamp, rounded to the specified precision. The precision argument behaves like [DATE\_TRUNC](#date_trunc). The function rounds up values to the specified precision. **Syntax** ```sql SQL theme={null} ROUND(time, precision) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `precision` | `CHAR` | The timestamp unit used to round the returned value.
For the `precision` input, acceptable values are:
`NANOSECOND`
`MICROSECOND`
`MILLISECOND`
`SECOND`
`MINUTE`
`HOUR`
`DAY`
`WEEK`
`MONTH`
`QUARTER`
`YEAR`
`DECADE`
`CENTURY`
`MILLENNIUM` | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function rounds to the specified precision. | **Example** ```sql SQL theme={null} SELECT ROUND(CURRENT_TIMESTAMP() ,'HOUR'); ``` *Output*: `2023-02-08 09:00:00.000` ### SECOND Extracts the seconds portion of a timestamp as an integer. **Syntax** ```sql SQL theme={null} SECOND(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ---------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `TIME` | A time value, specified as a timestamp or date, which the function uses to return the seconds value. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-02-28 13:17:30.243`. ```sql SQL theme={null} SELECT SECOND(CURRENT_TIMESTAMP()); ``` *Output*: `30` ### TIMESTAMP\_TO\_NANOS Convert timestamp into nanoseconds after epoch as `BIGINT`. **Syntax** ```sql SQL theme={null} TIMESTAMP_TO_NANOS(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------ | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function uses to return the nanoseconds after the epoch. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-02-28 13:20:12.373`. ```sql SQL theme={null} SELECT TIMESTAMP_TO_NANOS(CURRENT_TIMESTAMP()); ``` *Output*: `1677619212373708904` ### TO\_TIMESTAMP For information on using the TO\_TIMESTAMP conversion function, see the [TO\_TIMESTAMP](/formatting-functions#to_timestamp) page. ### USECS Returns the seconds part of a time value, including fractional parts, as an integer. Same as `EXTRACT(MICROSECONDS FROM time)`. **Syntax** ```sql SQL theme={null} USECS(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date. The function multiplies the seconds part of the value by 1,000,000. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-02-28 13:32:17.438`. ```sql SQL theme={null} SELECT USECS(CURRENT_TIMESTAMP()); ``` *Output*: `17438041` ### WEEK Returns the ISO-8601 week number, as an integer, of the specified timestamp or date value. The week starts on Monday, and the first week of a year contains January 4 of that year. For details, see the [ISO week date](https://en.wikipedia.org/wiki/ISO_week_date) definition. **Syntax** ```sql SQL theme={null} WEEK(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | -------------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function uses to return the week number. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-02-28 13:37:37.128`. ```sql SQL theme={null} SELECT WEEK(CURRENT_TIMESTAMP()); ``` *Output*: `9` ### YEAR Extracts the year portion of a timestamp or date as an integer. **Syntax** ```sql SQL theme={null} YEAR(time) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------- | -------------------------------------------------------------------------------------------- | | `time` | `TIMESTAMP` or `DATE` | A time value, specified as a timestamp or date, which the function uses to extract the year. | **Example** This example uses `CURRENT_TIMESTAMP` set to `2023-02-28 13:37:37.128`. ```sql SQL theme={null} SELECT YEAR(CURRENT_TIMESTAMP()); ``` *Output*: `2023` ## Advanced Date and Time Functions ### DATEADD This function adds a specified number value (as a signed integer) to a specified `datepart` of an input date value, and then returns that modified value. The data type of the returned value for this function is dynamic. The return type depends on the argument supplied for `date`. If the value for `date` is a string literal `date`, `DATEADD` returns a `datetime` value. If another valid input data type is supplied for `date`, `DATEADD` returns the same data type. `DATEADD` raises an error if the string literal seconds scale exceeds three decimal place positions (.nnn) or if the string literal contains the time zone offset part. **Syntax** ```sql SQL theme={null} DATEADD(datepart, number, date) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `datepart` | `CHAR` | The part of the date to which DATEADD adds an integer number.
See the Datepart Values table for a list of supported arguments. | | `number` | `INT` | An expression that can resolve to an `INT` that `DATEADD` adds to the specified `datepart` value of `date`.
`DATEADD` accepts user-defined variable values for number.
`DATEADD` truncates a specified `number` value that has a decimal fraction. In this situation, it does not round the number value. | | `date` | `TIME`, `TIMESTAMP`, `DATE` | A column expression, expression, string literal, or user-defined variable. A string literal value must resolve to a date or timestamp.
Use four-digit years to avoid ambiguity issues. | | ***datepart*** | **Abbreviations** | | -------------- | ----------------- | | `year` | yy, yyyy | | `quarter` | qq, q | | `month` | mm, m | | `dayofyear` | dy, y | | `day` | dd, d | | `week` | wk, ww | | `weekday` | dw, w | | `hour` | hh | | `minute` | mi, n | | `second` | ss, s | | `millisecond` | ms | | `microsecond` | mcs | | `nanosecond` | ns | `DATEADD` does not accept user-defined variable equivalents for the `datepart` arguments. **Examples** This example shows each result from incrementing different `datepart` values by 1. ```sql SQL theme={null} SELECT 'year', DATEADD(year,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'quarter', DATEADD(quarter,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'month', DATEADD(month,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'dayofyear', DATEADD(dayofyear,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'day', DATEADD(day,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'week', DATEADD(week,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'weekday', DATEADD(weekday,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'hour', DATEADD(hour,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'minute', DATEADD(minute,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'second', DATEADD(second,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'millisecond', DATEADD(millisecond,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'microsecond', DATEADD(microsecond,1, '2007-01-01 13:10:10.1111111') UNION ALL SELECT 'nanosecond', DATEADD(nanosecond,1, '2007-01-01 13:10:10.1111111'); ``` *Output* ```Text Text theme={null} week 2007-01-08 13:10:10.111111100 millisecond 2007-01-01 13:10:10.112111100 second 2007-01-01 13:10:11.111111100 day 2007-01-02 13:10:10.111111100 minute 2007-01-01 13:11:10.111111100 dayofyear 2007-01-02 13:10:10.111111100 quarter 2007-04-01 13:10:10.111111100 month 2007-02-01 13:10:10.111111100 hour 2007-01-01 14:10:10.111111100 weekday 2007-01-02 13:10:10.111111100 year 2008-01-01 13:10:10.111111100 nanosecond 2007-01-01 13:10:10.111111101 microsecond 2007-01-01 13:10:10.111112100 Fetched 13 rows ``` In this example, each statement increments `datepart` by a number large enough to increment the next higher unit of date or time. ```sql SQL theme={null} SELECT 'quarter', DATEADD(quarter,4,'2007-01-01 01:01:01.1111111') UNION ALL SELECT 'month', DATEADD(month,13,'2007-01-01 01:01:01.1111111') UNION ALL SELECT 'dayofyear', DATEADD(dayofyear,365,'2007-01-01 01:01:01.1111111') UNION ALL SELECT 'day', DATEADD(day,365,'2007-01-01 01:01:01.1111111') UNION ALL SELECT 'week', DATEADD(week,5,'2007-01-01 01:01:01.1111111') UNION ALL SELECT 'weekday', DATEADD(weekday,31,'2007-01-01 01:01:01.1111111') UNION ALL SELECT 'hour', DATEADD(hour,23,'2007-01-01 01:01:01.1111111') UNION ALL SELECT 'minute', DATEADD(minute,59,'2007-01-01 01:01:01.1111111') UNION ALL SELECT 'second', DATEADD(second,59,'2007-01-01 01:01:01.1111111'); ``` *Output* ```none Text theme={null} day 2008-01-01 01:01:01.111111100 weekday 2007-02-01 01:01:01.111111100 second 2007-01-01 01:02:00.111111100 month 2008-02-01 01:01:01.111111100 quarter 2008-01-01 01:01:01.111111100 dayofyear 2008-01-01 01:01:01.111111100 week 2007-02-05 01:01:01.111111100 hour 2007-01-02 00:01:01.111111100 minute 2007-01-01 02:00:01.111111100 Fetched 9 rows ``` ### DATEDIFF This function returns an `INT` representing the difference between two date or time values, in a specified date or time unit. If only a time value is assigned to a date data type variable, `DATEDIFF` sets the value of the missing `datepart` to the default value: `1900-01-01`. If only a date value is assigned to a variable of a time or date data type, `DATEDIFF` sets the value of the missing time part to the default value: `00:00:00`. If either `startdate` or `enddate` have only a time part and the other only a `datepart`, `DATEDIFF` sets the missing time and `datepart` to the default values. If `startdate` and `enddate` have different date data types, and one has more time parts or fractional seconds precision than the other, `DATEDIFF` sets the missing parts of the other to 0. **Syntax** ```sql SQL theme={null} DATEDIFF(datepart, startdate, enddate) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `datepart` | `CHAR` | The units in which the function returns the difference between the `startdate` and `enddate`. Commonly used `datepart` units include `month` or `second`.
The `datepart` value cannot be specified in a variable, or in a quoted string like `'month'`.
If `datepart` has a DATE value, and `startdate` and `enddate` are both assigned a TIME value, then the function returns `0`.
See the Datepart Values table for a list of supported arguments. | | `startdate` | `TIME`, `TIMESTAMP`, `DATE` | A starting date or time value to determine the difference from the `enddate` value.
If only a time value is assigned, `DATEDIFF` sets the value of the missing `datepart` to the default value: `1900-01-01` | | `enddate` | `TIME`, `TIMESTAMP`, `DATE` | An ending date or time value to determine the difference from the `startdate` value.
If only a time value is assigned, `DATEDIFF` sets the value of the missing `datepart` to the default value: `1900-01-01` | | ***datepart*** | **Abbreviations** | | -------------- | ----------------- | | `year` | yy, yyyy | | `quarter` | qq, q | | `month` | mm, m | | `dayofyear` | dy, y | | `day` | dd, d | | `week` | wk, ww | | `weekday` | dw, w | | `hour` | hh | | `minute` | mi, n | | `second` | ss, s | | `millisecond` | ms | | `microsecond` | mcs | | `nanosecond` | ns | For `millisecond`, the maximum difference between `startdate` and `enddate` is 24 days, 20 hours, 31 minutes, and 23.647 seconds. For `second`, the maximum difference is 68 years, 19 days, 3 hours, 14 minutes, and 7 seconds. Each specific `datepart` name and abbreviations for that `datepart` name return the same value. **Example** The example shows how `DATEDIFF` calculates the difference between the two values based on the various `datepart` values. ```sql SQL theme={null} SELECT 'year', DATEDIFF(year, '2004-05-31 23:59:59.9999999', '2006-01-01 00:00:00.0000000') UNION ALL SELECT 'quarter', DATEDIFF(quarter, '2004-05-31 23:59:59.9999999', '2006-01-01 00:00:00.0000000') UNION ALL SELECT 'month', DATEDIFF(month, '2004-05-31 23:59:59.9999999', '2006-01-01 00:00:00.0000000') UNION ALL SELECT 'dayofyear', DATEDIFF(dayofyear, '2004-05-31 23:59:59.9999999', '2006-01-01 00:00:00.0000000') UNION ALL SELECT 'day', DATEDIFF(day, '2004-05-31 23:59:59.9999999', '2006-01-01 00:00:00.0000000') UNION ALL SELECT 'week', DATEDIFF(week, '2004-05-31 23:59:59.9999999', '2006-01-01 00:00:00.0000000') UNION ALL SELECT 'hour', DATEDIFF(hour, '2004-05-31 23:59:59.9999999', '2006-01-01 00:00:00.0000000') UNION ALL SELECT 'minute', DATEDIFF(minute, '2004-05-31 23:59:59.9999999', '2006-01-01 00:00:00.0000000') UNION ALL SELECT 'second', DATEDIFF(second, '2004-05-31 23:59:59.9999999', '2006-01-01 00:00:00.0000000'); ``` *Output* ```none Text theme={null} day 580 quarter 7 year 2 dayofyear 580 month 20 hour 13897 week 83 minute 833761 second 50025601 ``` ## Related Links [Time Zone Functions](/time-zone-functions) [Formatting Functions](/formatting-functions) # DBeaver Integration Source: https://docs.ocient.com/dbeaver-integration Learn how to integrate Ocient with DBeaver, enabling a powerful and intuitive interface for managing and analyzing your Ocient data. DBeaver is a free and open-source application for database administration and SQL querying. ## Prerequisites To use DBeaver with the System, you must have these software prerequisites: * Ocient System — Use latest version. * JDBC Driver — Use latest version. See the Ocient [repository](https://mvnrepository.com/artifact/com.ocient/ocient-jdbc4) for a list of all versions. To install, download the latest version of DBeaver from the [DBeaver Community](https://dbeaver.io/download/) for your respective operating system. Follow the installation steps and open DBeaver. ## Connect to Ocient You can connect to your database by following these steps: * In the main DBeaver window, select **New Database Connection** in the top left of the window. * In the New Database Connection window, find the Ocient driver. Select it, and then select **Next**. * Enter your database parameters with this information. | **Field** | **Description** | | --------------- | -------------------------------------------------------------------------------------------------- | | Host | The hostname or IP address of the SQL Node of your Ocient System. For example: `databasehost-sql0` | | Port | The port number for your connection. The default Ocient port is `4050`. | | Database/Schema | The name of the database for connection. For example: `my_db` | | User | The username associated with your database. | | Password | The password associated with your username. | * Select **Finish**. ## Update Ocient JDBC By default, DBeaver downloads the latest JDBC version available in the [Maven repository](https://mvnrepository.com/artifact/com.ocient/ocient-jdbc4). If you need to update the JDBC driver or load a specific version, follow these steps. * In the DBeaver window, select **Database** > **Driver Manager**. Selection of Driver Manager in the Database menu * Search among the list of driver options to find the Ocient driver, and select it. * With the Ocient driver highlighted, select **Edit**. Select the Ocient driver under Name in the Driver Manager window. * In the Edit window, select the **Libraries** tab. **Update the Driver to the Latest Version** To update the driver to the latest version, select **Download/Update**. Libraries tab of the Edit Driver Ocient window contains the Download/Update button. **Load a Specific JDBC Driver Version** * Double-click the `com.ocient:ocient-jdbc4:RELEASE` entry in the Libraries window. The Edit Maven Artifact window opens. * Enter the JDBC version in the **Version** field. Specify the specific JDBC version to load in the Declare Artifact Manually section by using the Version field. For accurate time values, you might need to configure the DBeaver time zone settings. * For systems, configure this setting in **Settings** > **User Interface**. * For systems, configure this setting in **Preferences** > **User Interface**. ## Related Links [Connect to Ocient](/connect-to-ocient) [Ocient Integrations](/ocient-integrations) # Deduplication in Data Pipelines Source: https://docs.ocient.com/deduplication-in-data-pipelines Ocient data pipelines deduplicate records by tracking load position and lifecycle, ensuring exactly-once delivery even when pipelines stop, resume, or update. ## Exactly Once Loading in Pipelines The job of a pipeline is to load the data assigned to it exactly once. A pipeline maintains its own position during a load and enforces deduplication logic. The lifecycle of a pipeline object in the database defines both the deduplication logic and load position. Because the pipeline, not the target tables, defines the load position and deduplication scope, if the system truncates target tables or if the system drops and recreates a target table, the pipeline continues from its last position. It does not reset to the beginning position of the load if you attempt to load data into a new table. Key benefits are: * Pipeline Events — During the life of a pipeline, you can stop and resume the pipeline without duplicating source data or changing the position in a load. * Pipeline Updates — You can modify pipelines using the `CREATE OR REPLACE` SQL statement to update the transforms in a pipeline but maintain the current position in the load. * Exactly Once Loading — In the event of transient system failures, the pipeline ensures exactly-once loading into the target table through replay and deduplication guarantees. Two concepts are critical to understand how controls which data is loaded in a pipeline and how Ocient pipelines ensure exactly-once loading. 1. Load Position 2. Deduplication Scope ## Load Position You can control the pipeline by starting, stopping, modifying, and resuming the pipeline. While running, each pipeline maintains its position in the overall load. When it has reached the end of all data, the pipeline completes and does not attempt to reload the data. To load the data again, you must create a new pipeline or drop and recreate the pipeline. * For an -based load, the position consists of using consumer group offsets appropriately to create checkpoints. * For File-based loads, file details are stored in the `sys.pipeline_files` system catalog table and the system updates the status of each file as the load progresses. In both cases, the load position defines where the pipeline resumes loading if stopped and resumed. ## Deduplication Scope The deduplication scope defines the conditions under which a pipeline does not cause the same row to appear twice in a target table. This deduplication is how a pipeline ensures that it is safe to replay data during loading. This situation is common in failover situations or when you stop and restart a pipeline. The deduplication scope is the unique combination of: 1. A pipeline object 2. A target table A pipeline guarantees that if it sends the same row to the same target table twice, the system only loads it into the table once. If you drop and recreate either the pipeline or the table, then the situation is a new deduplication scope. If you begin sending data to a new target table, this situation is a new deduplication scope. A row for the same record can appear in both the old and new tables. ## Restarts and Deduplication You can restart a pipeline without creating duplicate data in the target tables. However, there are some limitations and key assumptions for each data source. For example, in a file load, if you modify the contents of a file after the pipeline has started, then you can experience duplication of data or missed rows. For more details and key considerations, see [Restarting a File Load](/data-pipelines) and [Restarting a Kafka Load](/data-pipelines). ## Deduplication and DROP TABLE You can drop a table using the `DROP TABLE` SQL statement and then recreate it with the same name. In this case, an existing pipeline loads into the new table. When you drop a table and recreate it, these actions do not update the load position of the pipeline. When you restart the pipeline, it resumes from the load position of the pipeline, and all records from that position forward are loaded with a new deduplication scope. ## Deduplication and TRUNCATE TABLE You can truncate a table (using the `TRUNCATE TABLE` SQL statement) that is the target of a pipeline whether the pipeline is running or stopped. These actions do not update the load position of the pipeline or change the deduplication scope. The system deduplicates any data that was loaded before the truncate and replays after the truncate for any reason (e.g., restart or transient failure). Also, the system loads all new data. When you truncate the target tables and restart the pipeline, the Ocient System does not reload the data. ## Load Duplicate Data Sometimes, you might want to load the same data multiple times, but the pipeline load position and deduplication scope prevent this. If you want to load a second copy of the source data, you can follow one of these approaches: File-Based Pipelines * Drop and recreate the pipeline to reset the `sys.pipeline_files` system catalog table. * Create a second pipeline with a new name and the same configuration. Kafka-Based Pipelines * Drop the pipeline and recreate it with the same name, which defaults to the same consumer `group.id` but has a new deduplication scope. Then, reset the consumer group offsets using Kafka tools to the chosen starting point. * Create a new pipeline with a different name and load from the beginning of the topic using `auto.offset.reset`. ## Related Links [Data Pipelines DDL Reference](/data-pipelines) [START PIPELINE](/data-pipelines#start-pipeline) [Restart with File Loading](/data-pipelines) [Restart with Kafka Loading](/data-pipelines) # Discover Insights From System Catalog Tables Source: https://docs.ocient.com/discover-insights-from-system-catalog-tables Sample SQL queries on Ocient system catalog tables to track storage use, node status, running queries, segment sizes, and other operational metrics. You can query the System catalog tables to discover important information on system objects such as databases, segments, tables, and indexes. These statistics can be important for monitoring your system to ensure it operates as intended. However, querying these tables can be challenging because generating insights about your system can involve writing complex queries that involve merging, grouping, and filtering multiple system catalog tables. Here, you can find examples of useful queries that you can use as templates for discovering insights about your system. Most of these example queries use the `data_type_coverage` table available with the Ocient Simulator. For details, see [Ocient Simulator](/ocient-simulator). ## Segment System Catalog Tables For specific column definitions of the system catalog tables referenced in these examples, see the [Storage](/system-catalog#storage) section of the System Catalog page. ### Segment Size Check the size of segments and return insights about the segment storage allocation and the number of table rows contained in each segment. ```sql SQL theme={null} SELECT current_timestamp, sg.segment_type, t.name, (COUNT(*)) AS num_segments, (SUM(s.row_count)) AS total_rows, (AVG(s.row_count)) AS avg_rows_per_segment, (DOUBLE(SUM(s.segment_size)) / 1000000000) AS total_size_gb, (MAX(s.segment_size / 1000000)) AS max_size, (MIN(s.segment_size / 1000000)) AS min_size, (AVG(s.segment_size) / 1000000) AS avg_size_mb, (STDEV(DOUBLE(s.segment_size) / 1000000)) AS stdev_size_mb FROM sys.segment_groups AS sg JOIN sys.tables t ON t.id = sg.table_id JOIN sys.segments s ON s.segment_group_id = sg.id GROUP BY sg.segment_type, t.name ORDER BY total_size_gb DESC, total_rows DESC, sg.segment_type, t.name; ``` *Output* ```none Text theme={null} | "current_timestamp()" | "segment_type" | "name" | "num_segments" | "total_rows" | "avg_rows_per_segment" | "total_size_gb" | "max_size" | "min_size" | "avg_size_mb" | "stdev_size_mb" | |-------------------------|----------------|--------------------|----------------|--------------|------------------------|-----------------|------------|------------|--------------------|-------------------| | 2024-07-31 16:23:22.863 | TKT_SEGMENT | data_type_coverage | 3 | 100000 | 33333.333333333336 | 0.189542692 | 113 | 37 | 63.180897333333334 | 43.73338035824336 | ``` For information on the system catalog tables queried in this example, see [sys.segments](/system-catalog#sys-segments) and [sys.tables](/system-catalog#sys-tables). ### Page and Segment Information Check the details for pages and segments in the system. ```sql SQL theme={null} WITH cte AS ( SELECT COALESCE(v.schema, t.schema) AS schema, COALESCE(v.name, t.name) AS name, stored_segments.status, stored_segments.kind, SUM(CASE WHEN segments.segment_type = 'PAGE' THEN 1 ELSE 0 END) AS pagecount, SUM(CASE WHEN segments.segment_type = 'TKT_SEGMENT' THEN 1 ELSE 0 END) AS segcount, SUM(CASE WHEN segments.segment_type = 'PAGE' THEN segments.row_count ELSE 0 END) AS page_rows, SUM(CASE WHEN segments.segment_type = 'TKT_SEGMENT' THEN segments.row_count ELSE 0 END) AS seg_rows, MIN(CASE WHEN segments.segment_type = 'PAGE' THEN sg.begin_time ELSE NULL END) AS minpagetime, MAX(CASE WHEN segments.segment_type = 'PAGE' THEN sg.end_time ELSE NULL END) AS maxpagetime, MIN(CASE WHEN segments.segment_type = 'TKT_SEGMENT' THEN sg.begin_time ELSE NULL END) AS minsegtime, MAX(CASE WHEN segments.segment_type = 'TKT_SEGMENT' THEN sg.end_time ELSE NULL END) AS maxsegtime FROM sys.segments JOIN sys.stored_segments ON segments.segment_group_id = stored_segments.segment_group_id AND segments.ida_offset = stored_segments.ida_offset JOIN sys.segment_groups sg ON sg.id = segments.segment_group_id JOIN sys.tables AS t ON t.id = sg.table_id LEFT JOIN sys.views AS v ON v.global_dictionary_compression_table_id = t.id WHERE stored_segments.end_osn = 'OSN_INFINITY' GROUP BY 1, 2, stored_segments.status, stored_segments.kind ) SELECT schema, name, status, kind, pagecount, segcount, page_rows, seg_rows, page_rows + seg_rows AS tot_rows, CEILING(page_rows /(pagecount +.000001)) AS rows_per_page, CEILING(seg_rows /(segcount +.000001)) AS rows_per_seg, to_timestamp(minpagetime / 1000000000) AS minpagetime, to_timestamp(maxpagetime / 1000000000) AS maxpagetime, to_timestamp(minsegtime / 1000000000) AS minsegtime, to_timestamp(maxsegtime / 1000000000) AS maxsegtime, NOW() FROM cte ORDER BY schema, name; ``` *Output* ```none Text theme={null} | "schema" | "name" | "status" | "kind" | "pagecount" | "segcount" | "page_rows" | "seg_rows" | "tot_rows" | "rows_per_page" | "rows_per_seg" | "minpagetime" | "maxpagetime" | "minsegtime" | "maxsegtime" | "current_timestamp()" | |----------|--------------------|----------|--------|-------------|------------|-------------|------------|------------|-----------------|----------------|---------------|---------------|-------------------------|-------------------------|-------------------------| | loading | data_type_coverage | INTACT | DISK | 0 | 3 | 0 | 100000 | 100000 | 0.0 | 33334.0 | | | 1970-01-01 00:00:00.000 | 1970-01-01 00:00:00.000 | 2024-08-07 21:41:36.800 | ``` ### Storage Spaces on a System Check all storage spaces on your system. ```sql SQL theme={null} SELECT * FROM sys.storage_spaces; ``` *Output* ```none Text theme={null} | "id" | "name" | "is_system_storage_space" | "block_size" | "total_width" | "parity_width" | "parity_type" | "parity_cycles" | "page_replication" | |----------------------------------------|--------------------|---------------------------|--------------|---------------|----------------|---------------|-----------------|--------------------| | ce9f9c2e-d3fe-4101-a2b2-6f7f2580327b | ss0 | false | 4096 | 3 | 1 | XOR | 1 | 2 | | "7277656e-6465-6c20-7761-732068657265" | systemstoragespace | true | 4096 | 3 | 2 | REPLICATION | 1 | 3 | ``` ## Query Management System Catalog Tables For specific column definitions of the system catalog tables referenced in these examples, see the [Monitoring](/system-catalog#monitoring) section of the System Catalog page. ### Running Queries Check all queries currently running on the system. The output of the `sys.queries` table depends on what database you are logged into and your system role. For details, see [Query Visibility](/data-control-language-dcl-statement-reference#query-visibility). ```sql SQL theme={null} SELECT queries.query_id, queries.initial_priority, queries.effective_priority, queries.user, queries.total_time, queries.status, queries.rows_returned, queries.bytes_returned FROM sys.queries WHERE queries.sql NOT LIKE '%sys.queries%'; SELECT q.query_id, q.user, q.sql, q.status, "database_name", q.bytes_returned FROM sys.queries q; ``` *Output* ```none Text theme={null} | "query_id" | "user" | "sql" | "status" | "database_name" | "bytes_returned" | |----------------------------------------|--------------|------------------|----------|-----------------|------------------| | "66b27dde-17df-403a-82d7-65e34b6fd07f" | admin@system | "example_query" | RUNNING | test | 0 | ``` ### Recent Queries Check the most recent queries completed by the system. The output of the `sys.completed_queries` table depends on what database you are logged into and your system role. For details, see [Query Visibility](/data-control-language-dcl-statement-reference). ```sql SQL theme={null} SELECT user, timestamp_start, total_time, state, reason, sql FROM sys.completed_queries ORDER BY timestamp_start DESC LIMIT 10; ``` *Output* ```none Text theme={null} | "user" | "time_start" | "total_time" | "state" | "reason" | "sql" | | | | | | | | | | | | | | |--------------|---------------------------|--------------|---------|-----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|-------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|------------|---------|------------------------------------|----------|------------|---------|--------------------------------------------------------------------| | admin@system | "2024-08-06T21:51:29.284" | 84 | "00000" | The operation completed successfully | "with the_user_query_to_add_a_limit_to as (select user | time_start | total_time | state | reason | sql from sys.completed_queries order by time_start desc limit 100) select * from the_user_query_to_add_a_limit_to limit 200" | | | | | | | | | | admin@system | "2024-08-06T21:45:33.952" | 58 | "00000" | The operation completed successfully | "with the_user_query_to_add_a_limit_to as (select q.query_id | q.user | q.sql | q.status | ""database_name"" | q.bytes_returned from sys.queries q) select * from the_user_query_to_add_a_limit_to limit 200" | | | | | | | | | | admin@system | "2024-08-06T21:41:04.521" | 205 | "00000" | The operation completed successfully | "with the_user_query_to_add_a_limit_to as (select sg.segment_type | ss.kind | ss.status | t.name | (ss.end_osn = 'OSN_INFINITY') active | count(distinct ss.storage_id) num_segments from sys.stored_segments ss join sys.segment_groups sg on sg.id = ss.segment_group_id join sys.tables t on t.id = sg.table_id where ss.status <> 'INTACT' group by sg.segment_type | ss.kind | ss.status | t.name | active order by num_segments desc | ss.kind | ss.status | t.name | active) select * from the_user_query_to_add_a_limit_to limit 200" | | admin@system | "2024-08-06T21:38:19.740" | 90 | "00000" | The operation completed successfully | with the_user_query_to_add_a_limit_to as (select * from sys.stored_segments) select * from the_user_query_to_add_a_limit_to limit 200 | | | | | | | | | | | | | | | admin@system | "2024-08-06T21:37:35.181" | 5 | "42703" | The reference to referenced column is not valid (The column 'ss.kind' is being referenced incorrectly. Maybe a missing GROUP BY?) | "with the_user_query_to_add_a_limit_to as (select ss.kind | ss.status | (ss.end_osn = 'OSN_INFINITY') active | count(distinct ss.storage_id) num_segments from sys.stored_segments ss) select * from the_user_query_to_add_a_limit_to limit 200" | | | | | | | | | | | | admin@system | "2024-08-06T21:37:33.710" | 5 | "42703" | The reference to referenced column is not valid (The column 'ss.kind' is being referenced incorrectly. Maybe a missing GROUP BY?) | "with the_user_query_to_add_a_limit_to as (select ss.kind | ss.status | (ss.end_osn = 'OSN_INFINITY') active | count(distinct ss.storage_id) num_segments from sys.stored_segments ss) select * from the_user_query_to_add_a_limit_to limit 200" | | | | | | | | | | | | admin@system | "2024-08-06T21:36:03.692" | 11 | "42703" | The reference to referenced column is not valid (The column 'ss.kind' is being referenced incorrectly. Maybe a missing GROUP BY?) | "with the_user_query_to_add_a_limit_to as (select ss.kind | ss.status | (ss.end_osn = 'OSN_INFINITY') active | count(distinct ss.storage_id) num_segments from sys.stored_segments ss) select * from the_user_query_to_add_a_limit_to limit 200" | | | | | | | | | | | | admin@system | "2024-08-06T21:35:51.301" | 4 | "42703" | The reference to column 't.name' is not valid | "with the_user_query_to_add_a_limit_to as (select ss.kind | ss.status | t.name | (ss.end_osn = 'OSN_INFINITY') active | count(distinct ss.storage_id) num_segments from sys.stored_segments ss) select * from the_user_query_to_add_a_limit_to limit 200" | | | | | | | | | | | admin@system | "2024-08-06T21:35:10.331" | 5 | "42703" | The reference to column 'sg.segment_type' is not valid | "with the_user_query_to_add_a_limit_to as (select sg.segment_type | ss.kind | ss.status | t.name | (ss.end_osn = 'OSN_INFINITY') active | count(distinct ss.storage_id) num_segments from sys.stored_segments ss) select * from the_user_query_to_add_a_limit_to limit 200" | | | | | | | | | | admin@system | "2024-08-06T21:34:42.970" | 208 | "00000" | The operation completed successfully | "with the_user_query_to_add_a_limit_to as (select sg.segment_type | ss.kind | ss.status | t.name | (ss.end_osn = 'OSN_INFINITY') active | count(distinct ss.storage_id) num_segments from sys.stored_segments ss join sys.segment_groups sg on sg.id = ss.segment_group_id join sys.tables t on t.id = sg.table_id where ss.status <> 'INTACT' group by sg.segment_type | ss.kind | ss.status | t.name | active order by num_segments desc | ss.kind | ss.status | t.name | active) select * from the_user_query_to_add_a_limit_to limit 200" | ``` ## Database and Table System Catalog Tables For specific column definitions of the system catalog tables used in these examples, see the [Databases](/system-catalog#databases) section of the System Catalog page. ### Database Sizes Check the total size of all databases on your system. ```sql SQL theme={null} SELECT TO_CHAR(SUM(sp.size), '999,999,999,999,999') AS totsize, d.name FROM sys.segment_parts AS sp JOIN sys.segment_groups AS sg ON sp.segment_group_id = sg.id JOIN sys.tables AS t ON sg.table_id = t.id JOIN sys.databases AS d ON t.database_id = d.id GROUP BY d.name ORDER BY 1 DESC; ``` *Output* ```none Text theme={null} | "totsize" | "name" | |-----------|--------| | 204276004 | test | ``` ### Table Sizes Check the total size of all tables on the system. ```sql SQL theme={null} WITH cte AS ( SELECT sg.id AS segment_group_id, sp.name AS segment_name, sp.part_type AS partition_type, sp.size AS segment_size, sg.segment_type, sg.status, t.schema, t.name FROM sys.segment_parts AS sp INNER JOIN sys.segment_groups AS sg ON sp.segment_group_id = sg.id INNER JOIN sys.tables AS t ON sg.table_id = t.id ), ctefinal AS ( SELECT schema, name, SUM(segment_size) AS total_table_size FROM cte GROUP BY schema, name ) SELECT SCHEMA, name, total_table_size, SUM(total_table_size) OVER( ORDER BY total_table_size DESC ) AS running_total FROM ctefinal ORDER BY total_table_size DESC; ``` *Output* ```none Text theme={null} | "schema" | "name" | "total_table_size" | "running_total" | |----------|--------------------|--------------------|-----------------| | loading | data_type_coverage | 204276004 | 204276004 | ``` ### Table Statistics This query provides a breakdown of table statistics and partition information. ```sql SQL theme={null} WITH cte1 AS ( SELECT sg.id AS segment_group_id, sp.name AS segment_name, sp.part_type AS partition_type, sp.size AS segment_size, sg.segment_type, sg.status, t.schema, t.name FROM sys.segment_parts AS sp INNER JOIN sys.segment_groups AS sg ON sp.segment_group_id = sg.id INNER JOIN sys.tables AS t ON sg.table_id = t.id ), cte AS ( SELECT COUNT(*) AS num_segments, SUM(segment_size) AS partition_size, partition_type, SUM(SUM(segment_size)) OVER(PARTITION BY schema, name) AS total_table_size, SUM(COUNT(*)) OVER(PARTITION BY schema, name) AS total_table_segments, schema, name FROM cte1 GROUP BY partition_type, schema, name ) SELECT name, num_segments, partition_size, partition_type, CAST(FLOAT(partition_size)/(total_table_size + 1) * 100 AS DECIMAL(6,3)) AS percent_of_total, total_table_size, total_table_segments FROM cte ORDER BY name, partition_size DESC; ``` *Output* ```none Text theme={null} | "name" | "num_segments" | "partition_size" | "partition_type" | "percent_of_total" | "total_table_size" | "total_table_segments" | |--------------------|----------------|------------------|------------------------|--------------------|--------------------|------------------------| | data_type_coverage | 3 | 113397760 | parity_data | 59.827 | 189542692 | 42 | | data_type_coverage | 3 | 75599872 | data | 39.885 | 189542692 | 42 | | data_type_coverage | 3 | 172032 | skip_lists | 0.091 | 189542692 | 42 | | data_type_coverage | 3 | 172032 | copy_skip_lists | 0.091 | 189542692 | 42 | | data_type_coverage | 3 | 90112 | parity_stats | 0.048 | 189542692 | 42 | | data_type_coverage | 3 | 59210 | stats | 0.031 | 189542692 | 42 | | data_type_coverage | 3 | 12288 | table_of_contents | 0.006 | 189542692 | 42 | | data_type_coverage | 3 | 12288 | parity_index | 0.006 | 189542692 | 42 | | data_type_coverage | 3 | 12288 | copy_table_of_contents | 0.006 | 189542692 | 42 | | data_type_coverage | 3 | 8192 | index | 0.004 | 189542692 | 42 | | data_type_coverage | 3 | 4096 | parity_summary_stats | 0.002 | 189542692 | 42 | | data_type_coverage | 3 | 906 | column_metadata | 0.000 | 189542692 | 42 | | data_type_coverage | 3 | 906 | copy_column_metadata | 0.000 | 189542692 | 42 | | data_type_coverage | 3 | 710 | summary_stats | 0.000 | 189542692 | 42 | ``` For information on the system catalog tables queried in this example, see [sys.segment\_groups](/system-catalog#sys-segment_groups) and [sys.segment\_parts](/system-catalog#sys-segment_parts). ### Compression Check the information on the compression for each column in the specified table. ```sql SQL theme={null} SELECT r."table", c.NAME, r.is_rle_enabled, r.is_ne_enabled, r.raw_size, r.compressed_size, r.is_deltadelta_enabled, r.num_deltadelta_blocks, r.num_rle_blocks, r.num_nerle_blocks, r.num_uncompressed_blocks, r.num_total_blocks FROM sys.columns c, ( SELECT t.name AS "table", t.id, cci.is_deltadelta_enabled, cci.is_rle_enabled, cci.is_ne_enabled, cci.raw_size, cci.compressed_size, cci.num_deltadelta_blocks, cci.num_rle_blocks, cci.num_nerle_blocks, cci.num_uncompressed_blocks, cci.num_total_blocks, cci.ordinal FROM sys.tables AS t, sys.columns_compression_info AS cci WHERE t.id = cci.table_id AND t.name = 'my_table' ) AS r WHERE r.id = c.table_id AND r.ordinal = c.ordinal; ``` *Output* ```none Text theme={null} | "table" | "name" | "is_rle_enabled" | "is_ne_enabled" | "raw_size" | "compressed_size" | "is_deltadelta_enabled" | "num_deltadelta_blocks" | "num_rle_blocks" | "num_nerle_blocks" | "num_uncompressed_blocks" | "num_total_blocks" | |--------------------|---------------|------------------|-----------------|------------|-------------------|-------------------------|-------------------------|------------------|--------------------|---------------------------|--------------------| | data_type_coverage | col_double | true | true | 900000 | 905216 | false | 0 | 0 | 0 | 220 | 220 | | data_type_coverage | col_int | true | true | 500000 | 507904 | true | 0 | 0 | 0 | 123 | 123 | | data_type_coverage | col_tinyint | true | true | 200000 | 204800 | true | 0 | 0 | 0 | 49 | 49 | | data_type_coverage | col_date | true | true | 500000 | 294912 | true | 71 | 0 | 0 | 0 | 71 | | data_type_coverage | col_decimal | true | true | 1100000 | 1105920 | false | 0 | 0 | 0 | 269 | 269 | | data_type_coverage | col_ip | true | true | 1700000 | 1712128 | false | 0 | 0 | 0 | 417 | 417 | | data_type_coverage | col_point | true | true | 1700000 | 1712128 | false | 0 | 0 | 0 | 417 | 417 | | data_type_coverage | col_timestamp | true | true | 900000 | 905216 | true | 0 | 0 | 0 | 220 | 220 | | data_type_coverage | col_bigint | true | true | 900000 | 905216 | true | 0 | 0 | 0 | 220 | 220 | | data_type_coverage | col_binary | true | true | 300000 | 307200 | false | 0 | 0 | 0 | 74 | 74 | | data_type_coverage | col_boolean | true | true | 200000 | 106496 | true | 19 | 0 | 0 | 6 | 25 | | data_type_coverage | col_ipv4 | true | true | 500000 | 507904 | true | 0 | 0 | 0 | 123 | 123 | | data_type_coverage | col_time | true | true | 900000 | 905216 | true | 0 | 0 | 0 | 220 | 220 | | data_type_coverage | col_float | true | true | 500000 | 507904 | false | 0 | 0 | 0 | 123 | 123 | | data_type_coverage | col_uuid | true | true | 1700000 | 1712128 | false | 0 | 0 | 0 | 417 | 417 | | data_type_coverage | col_smallint | true | true | 300000 | 307200 | true | 0 | 0 | 0 | 74 | 74 | ``` For information on the system catalog table queried in this example, see [sys.columns\_compression\_info](/system-catalog#sys-columns_compression_info). ### Compression for the Entire Table This query summarizes the total compression of the specified table. In the example, replace the table name and schema name accordingly. ```sql SQL theme={null} SELECT SUM(raw_size), SUM(compressed_size) FROM ( SELECT t.name, c.name AS colname, co.raw_size, co.compressed_size, SUM(co.compressed_size) OVER(PARTITION BY t.schema, t.name) AS table_size, 100 -((100 * co.compressed_size) / co.raw_size) AS col_compress_percent, DECIMAL( FLOAT(co.compressed_size) * 100 / SUM(co.compressed_size) OVER(PARTITION BY t.schema, t.name), 3, 1 ) AS percent_of_table FROM ( SELECT table_id, ordinal, raw_size, compressed_size FROM sys.columns_compression_info UNION ALL SELECT table_id, ordinal, raw_size, compressed_size FROM sys.vl_columns_compression_info ) AS co JOIN sys.tables AS t ON co.table_id = t.id JOIN sys.columns AS c ON c.table_id = t.id AND co.ordinal = c.ordinal WHERE t.name = 'data_type_coverage' AND t.schema = 'loading' ORDER BY co.ordinal ) AS abc; ``` *Output* ```none Text theme={null} | "sum(raw_size)" | "sum(compressed_size)" | |-----------------|------------------------| | 73788914 | 73592306 | ``` For information on the system catalog table queried in this example, see [sys.columns\_compression\_info](/system-catalog). ### Index Size Check the total size of each index across the different segment groups. In this example, the table has three secondary indexes added before loading data. ```sql SQL theme={null} SELECT p.name AS uuid, p.part_type, i.name AS index_name, SUM(p.size) AS sum_size FROM ( SELECT * FROM sys.segment_parts WHERE part_type LIKE 'index' ) AS p INNER JOIN sys.indexes i ON p.name = i.id GROUP BY 1,2,3 ORDER BY 1,2; ``` *Output* ```none Text theme={null} | "uuid" | "part_type" | "index_name" | "sum_size" | |----------------------------------------|--------------|--------------|------------| | "669cbcdf-772f-4c97-b250-0cc9e573cd7b" | index | idx01 | 2527232 | | cccf6f5e-96a8-4621-b947-f318d120f6a0 | index | idx03 | 180224 | | d75ae71a-207e-4723-a86c-ecffc798c7d0 | index | idx02 | 3190784 | ``` For information on the system catalog tables queried in this example, see [sys.indexes](/system-catalog#sys-indexes) and [sys.segment\_parts](/system-catalog). ### Column Sizes This query finds the size of all columns for the specified table. Before running this query on your system, replace `table_name` in the example. ```sql SQL theme={null} WITH cte AS ( SELECT table_id, ordinal, compressed_size FROM sys.columns_compression_info UNION ALL SELECT table_id, ordinal, compressed_size FROM sys.vl_columns_compression_info ) SELECT ( SELECT t.name FROM sys.tables AS t WHERE t.id = cte.table_id ) AS table_name, ordinal, compressed_size, ( SELECT name FROM sys.columns AS c WHERE c.table_id = cte.table_id AND c.ordinal = cte.ordinal ) AS name, DOUBLE(compressed_size) / SUM(compressed_size) OVER() AS percent_of_total FROM cte WHERE table_id IN ( SELECT id FROM sys.tables WHERE name LIKE 'table_name' ) ORDER BY compressed_size; ``` *Output* ```none Text theme={null} | "table_name" | "ordinal" | "compressed_size" | "name" | "percent_of_total" | |--------------------|-----------|-------------------|----------------|-----------------------| | data_type_coverage | 25 | 8192 | | 0.0001082748327179 | | data_type_coverage | 21 | 8192 | | 0.0001082748327179 | | data_type_coverage | 2 | 102400 | col_boolean | 0.0013534354089736837 | | data_type_coverage | 18 | 204800 | col_tinyint | 0.0027068708179473675 | | data_type_coverage | 19 | 266240 | | 0.003518932063331578 | | data_type_coverage | 4 | 294912 | col_date | 0.0038978939778442096 | | data_type_coverage | 12 | 307200 | col_smallint | 0.004060306226921052 | | data_type_coverage | 1 | 307200 | col_binary | 0.004060306226921052 | | data_type_coverage | 10 | 507904 | col_ipv4 | 0.0067130396285094715 | | data_type_coverage | 7 | 507904 | col_float | 0.0067130396285094715 | | data_type_coverage | 8 | 507904 | col_int | 0.0067130396285094715 | | data_type_coverage | 3 | 699370 | col_char | 0.009243673066151615 | | data_type_coverage | 23 | 783147 | col_varbinary | 0.010350965627260875 | | data_type_coverage | 6 | 905216 | col_double | 0.011964369015327365 | | data_type_coverage | 17 | 905216 | col_timestamp | 0.011964369015327365 | | data_type_coverage | 16 | 905216 | col_time | 0.011964369015327365 | | data_type_coverage | 0 | 905216 | col_bigint | 0.011964369015327365 | | data_type_coverage | 5 | 1105920 | col_decimal | 0.014617102416915785 | | data_type_coverage | 13 | 1712128 | col_point | 0.022629440038039995 | | data_type_coverage | 22 | 1712128 | col_uuid | 0.022629440038039995 | | data_type_coverage | 11 | 1712128 | col_ip | 0.022629440038039995 | | data_type_coverage | 20 | 1784388 | | 0.02358451076706771 | | data_type_coverage | 24 | 1788957 | col_varchar | 0.023644899891907562 | | data_type_coverage | 14 | 8500000 | col_linestring | 0.11234571265894837 | | data_type_coverage | 9 | 21477958 | col_int_array | 0.28387723505517193 | | data_type_coverage | 15 | 27739482 | col_polygon | 0.3666366910682436 | ``` This output is based on a query using the table `data_type_coverage`. ### Column Cardinality This query finds the cardinality of each column in the specified table. Knowing the cardinality is useful for determining which compression scheme to use for a column. ```sql SQL theme={null} SELECT t.schema, t.name, c.name, c.ordinal, cc.cardinality FROM sys.column_cardinalities AS cc LEFT JOIN sys.tables AS t ON cc.table_id = t.id LEFT JOIN sys.columns AS c ON cc.column_id = c.id WHERE t.name LIKE 'data_type_coverage' ORDER BY c.ordinal; ``` *Output* ```none Text theme={null} | "schema" | "name" | "name_1" | "ordinal" | "cardinality" | |----------|--------------------|----------------|-----------|---------------| | loading | data_type_coverage | col_bigint | 0 | 98551 | | loading | data_type_coverage | col_binary | 1 | 48048 | | loading | data_type_coverage | col_boolean | 2 | 2 | | loading | data_type_coverage | col_char | 3 | 95206 | | loading | data_type_coverage | col_date | 4 | 29634 | | loading | data_type_coverage | col_decimal | 5 | 20059 | | loading | data_type_coverage | col_double | 6 | 53546 | | loading | data_type_coverage | col_float | 7 | 75680 | | loading | data_type_coverage | col_int | 8 | 99319 | | loading | data_type_coverage | col_int_array | 9 | 5000855 | | loading | data_type_coverage | col_int_array | 9 | 99899 | | loading | data_type_coverage | col_ipv4 | 10 | 99908 | | loading | data_type_coverage | col_ip | 11 | 99875 | | loading | data_type_coverage | col_smallint | 12 | 51227 | | loading | data_type_coverage | col_point | 13 | 100000 | | loading | data_type_coverage | col_linestring | 14 | 100000 | | loading | data_type_coverage | col_polygon | 15 | 99153 | | loading | data_type_coverage | col_time | 16 | 99902 | | loading | data_type_coverage | col_timestamp | 17 | 99829 | | loading | data_type_coverage | col_tinyint | 18 | 256 | | loading | data_type_coverage | col_uuid | 22 | 99892 | | loading | data_type_coverage | col_varbinary | 23 | 68834 | | loading | data_type_coverage | col_varchar | 24 | 93258 | | loading | data_type_coverage | | | 10410 | | loading | data_type_coverage | | | 2 | | loading | data_type_coverage | | | 1 | | loading | data_type_coverage | | | 93333 | | loading | data_type_coverage | | | 1 | ``` For information on the system catalog table queried in this example, see [sys.column\_cardinalities](/system-catalog#sys-column_cardinalities). ## Cluster and Node System Catalog Tables For specific column definitions of the system catalog tables referenced in these examples, see the [System](/system-catalog#system) section of the System Catalog page. ### Nodes Present on a System Check the status of all nodes in the system. ```sql SQL theme={null} SELECT name, status FROM sys.nodes; ``` *Output* ```none Text theme={null} | "name" | "status" | |-------------|----------| | foundation2 | ACCEPTED | | sql | ACCEPTED | | loader | ACCEPTED | | foundation1 | ACCEPTED | | foundation0 | ACCEPTED | ``` ### Nodes Roles on a System Check the roles of each node in the system. ```sql SQL theme={null} SELECT n.name, n.status, STRING_AGG(DISTINCT c.name, ', ' ORDER BY c.name) in_clusters, STRING_AGG(DISTINCT sr.service_role_type, ', ' ORDER BY sr.service_role_type) with_roles FROM sys.nodes n LEFT JOIN sys.node_clusters nc ON nc.node_id = n.id LEFT JOIN sys.clusters c ON c.id = nc.cluster_id LEFT JOIN sys.service_roles sr ON sr.node_id = n.id GROUP BY n.name, n.status ORDER BY n.name; ``` *Output* ```none Text theme={null} | "name" | "status" | "in_clusters" | "with_roles" | | | | |-------------|----------|---------------------|-------------------------|----------------|-------------|--------------| | foundation0 | ACCEPTED | "foundation_cluster | foundation_cluster-vm" | "health | lts | operatorvm" | | foundation1 | ACCEPTED | "foundation_cluster | foundation_cluster-vm" | "health | lts | operatorvm" | | foundation2 | ACCEPTED | "foundation_cluster | foundation_cluster-vm" | "health | lts | operatorvm" | | loader | ACCEPTED | | "health | streamloader" | | | | sql | ACCEPTED | initial-vm | "admin | health | operatorvm | sql" | ``` ### Node Status Check the operational status, software versions, and assigned roles of your nodes. Possible status values include `ACTIVE`, `STARTING`, `STOPPING`, `ERROR`, `UNKNOWN`, or `UNREACHABLE`. ```sql SQL theme={null} SELECT name, operational_status, software_version, ARRAY_AGG(service_role_type) FROM sys.node_status AS ns LEFT JOIN sys.nodes AS n ON ns.node_id = id LEFT JOIN sys.service_roles AS sr ON sr.node_id = n.id GROUP BY name, operational_status, software_version; ``` *Output* ```none Text theme={null} | name | operational_status | software_version | array_agg(service_role_type:) | |-------------|--------------------|------------------|-------------------------------| | loader | Active | 24.0.0 | ['health','streamloader'] | | foundation2 | Active | 24.0.0 | ['health','lts','operatorvm'] | ``` ### Foundation Nodes on a Specific Cluster Check for all Foundation Nodes on your system clusters. ```sql SQL theme={null} SELECT n.name, r.name FROM sys.nodes AS n, ( SELECT * FROM sys.clusters AS c, sys.node_clusters AS nc WHERE c.cluster_type = 'Foundation' AND c.id = nc.cluster_id ) AS r WHERE r.node_id = n.id; ``` *Output* ```none Text theme={null} | foundation2 | foundation_cluster | |-------------|--------------------| | foundation0 | foundation_cluster | | foundation1 | foundation_cluster | ``` ### Drive Status Check the status of all drives assigned to nodes in the system. ```sql SQL theme={null} SELECT n.name AS node_name, s.node_id, s.id AS serial_number, s.pci_address, s.device_status, s.device_model FROM sys.nodes AS n JOIN sys.storage_device_status AS s ON n.id = s.node_id; ``` *Output* ```none Text theme={null} | "node_name" | "node_id" | "serial_number" | "pci_address" | "device_status" | "device_model" | |-------------|----------------------------------------|----------------------------------------|----------------------------------------------------------|-----------------|----------------| | foundation1 | "6a96b307-57bc-477f-952c-e6cab38a7922" | "72062649-396c-4424-9da5-c90cf0589e00" | /var/opt/ocient/72062649-396c-4424-9da5-c90cf0589e00.dat | "" | NON-NVME DRIVE | | foundation1 | "6a96b307-57bc-477f-952c-e6cab38a7922" | ee2f0013-c7b7-4750-833e-3d717af85001 | /var/opt/ocient/ee2f0013-c7b7-4750-833e-3d717af85001.dat | "" | NON-NVME DRIVE | | loader | "83f0ecf9-9a31-4a56-85b9-7eb610447185" | ff3186fb-502f-417c-bcab-8b7aa3de8800 | /var/opt/ocient/ff3186fb-502f-417c-bcab-8b7aa3de8800.dat | "" | NON-NVME DRIVE | | loader | "83f0ecf9-9a31-4a56-85b9-7eb610447185" | "11ff09e2-a6c1-4f76-ae96-3a97eaf02a01" | /var/opt/ocient/11ff09e2-a6c1-4f76-ae96-3a97eaf02a01.dat | "" | NON-NVME DRIVE | | foundation0 | "9330a0b3-b3b7-4503-949c-043b196c0cc4" | "6156962f-fcf6-4299-bbd7-2618fc6f1d00" | /var/opt/ocient/6156962f-fcf6-4299-bbd7-2618fc6f1d00.dat | "" | NON-NVME DRIVE | | foundation0 | "9330a0b3-b3b7-4503-949c-043b196c0cc4" | e6a4b24f-0d49-4704-b7ce-18af163c0701 | /var/opt/ocient/e6a4b24f-0d49-4704-b7ce-18af163c0701.dat | "" | NON-NVME DRIVE | | sql | "1198a86f-549d-4736-ac31-1ba54a2b02e7" | "37203a6c-9deb-47cf-8194-95c4eecd7300" | /var/opt/ocient/37203a6c-9deb-47cf-8194-95c4eecd7300.dat | "" | NON-NVME DRIVE | | sql | "1198a86f-549d-4736-ac31-1ba54a2b02e7" | "0921d62f-a31c-4681-9a56-82cb1450a401" | /var/opt/ocient/0921d62f-a31c-4681-9a56-82cb1450a401.dat | "" | NON-NVME DRIVE | | foundation2 | babadc9c-f2a3-4512-ab9a-bab3fd88a544 | "842881df-b71c-4cfc-847b-19517e8b2800" | /var/opt/ocient/842881df-b71c-4cfc-847b-19517e8b2800.dat | "" | NON-NVME DRIVE | | foundation2 | babadc9c-f2a3-4512-ab9a-bab3fd88a544 | fc0e490a-a295-491d-b6a6-8b3c153a7301 | /var/opt/ocient/fc0e490a-a295-491d-b6a6-8b3c153a7301.dat | "" | NON-NVME DRIVE | ``` ## Roles, Privileges, and Service Class System Catalog Tables For specific column definitions of the system catalog tables referenced in these examples, see the [User Management](/system-catalog#user-management) section of the System Catalog page. ### Group Assignment for Users Check for all users on the system and their corresponding assigned group. ```sql SQL theme={null} SELECT g.name AS group_name, r.user_name FROM sys.groups AS g, ( SELECT u.user_name, ug.group_id FROM sys.users AS u, sys.user_groups AS ug WHERE u.id = ug.user_id ) AS r ORDER BY g.name ASC; ``` *Output* ```none Text theme={null} | group_name | user_name | |------------|-----------| | DBA | Humphrey | ``` ### User Roles Check for all users on the system and their assigned role. ```sql SQL theme={null} SELECT rl.name AS role_name, r.user_name FROM sys.roles AS rl, ( SELECT u.user_name, ur.role_id FROM sys.users AS u, sys.user_roles AS ur WHERE u.id = ur.user_id ) AS r ORDER BY rl.name ASC; ``` *Output* ```none Text theme={null} | role_name | user_name | |------------------------|-----------| | system administrator | Humphrey | | system analyst | Grimey | | database administrator | Charles | | database analyst | Edna | ``` ### User Privileges Check the privileges for the specific table name. Before running on your system, replace `username` in the example. ```sql SQL theme={null} SELECT t.schema AS db_name, t.name AS table_name, r.privilege FROM sys.tables AS t, ( SELECT object_type, object_id, privilege FROM sys.PRIVILEGES WHERE grantee = 'username' ) AS r WHERE r.object_id = t.id ORDER BY t.schema, t.name ASC; ``` *Output* ```none Text theme={null} | db_name | table_name | privilege | |----------|--------------------|-------------| | loading | data_type_coverage | VIEW | | loading | data_type_coverage | SELECT | ``` ### Privileges for the Specific Table Check the granted privileges across all database objects of the specific type, such as `GROUP`, `TABLE`, or `DATABASE`. In this example, the query checks for `TABLE` privileges. ```sql SQL theme={null} SELECT p.grantee, t.schema, t.name, p.privilege FROM sys.tables AS t, sys.privileges AS p WHERE object_type = 'TABLE' AND p.object_id = t.id; ``` *Output* ```none Text theme={null} | admin@system | admin@system | example_table | DELETE | |--------------|--------------|--------------------|---------| | admin@system | admin@system | example_table | DROP | | admin@system | admin@system | example_table | SYSAUTH | | admin@system | admin@system | example_table | LOAD | | admin@system | admin@system | example_table | INSERT | | admin@system | admin@system | example_table | ALTER | | admin@system | admin@system | example_table | SELECT | | admin@system | admin@system | example_table | VIEW | ``` ### Groups Assigned to Service Classes Check for the service class assignment for all groups. ```sql SQL theme={null} SELECT g.name, sc.name FROM sys.groups AS g, sys.service_classes AS sc WHERE g.service_class_id = sc.id; ``` *Output* ```none Text theme={null} | group | sc_name | |---------|---------------| | analyst | high_priority | | dbadmin | low_priority | ``` ### Service Class Settings Check for the settings for the specified service class. Before running on your system, replace the `high priority` service class name in the example. For details about service class settings, see [Workload Management and Service Classes](/workload-management-and-service-classes). ```sql SQL theme={null} SELECT * FROM sys.groups AS g, sys.service_classes AS sc WHERE g.service_class_id = sc.id AND sc.name = 'high_priority'; ``` *Output* ```none Text theme={null} | id | name | database_id | service_class_id | id_1 | database_id_1 | name_1 | max_temp_disk_usage | max_elapsed_time | max_concurrent_queries | max_rows_returned | scheduling_priority | cache_max_bytes | cache_max_time | max_elapsed_time_for_caching | max_columns_in_result_set | priority_adjustment_factor | priority_adjustment_time | min_priority | max_priority | statement_text | statement_text_matcher_type | half_parallelism | load_balance_shuffle | parallelism | memory_optimal_strategy | |--------------------------------------|---------|--------------------------------------|--------------------------------------|--------------------------------------|--------------------------------------|---------------|---------------------|------------------|------------------------|-------------------|---------------------|-----------------|----------------|------------------------------|---------------------------|----------------------------|--------------------------|--------------|--------------|----------------|-----------------------------|------------------|----------------------|-------------|-------------------------| | e9c92a1b-32a8-4f88-ab72-485ff8b24f53 | analyst | e80010d7-6f26-438c-8461-11af309ed8a3 | 83b76f87-6634-4cc9-8252-9300caeefdf1 | 83b76f87-6634-4cc9-8252-9300caeefdf1 | e80010d7-6f26-438c-8461-11af309ed8a3 | high_priority | 80 | 100 | 10 | 100 | 5 | 1,000 | 25 | 50 | -1 | 0 | 0 | 0 | -1 | | | | | | false | ``` ### Service Classes for the Specific User Check for the service class of the specific user. Before running on your system, replace the `jmack@test` username in this example with the specific username that you want to query. ```sql SQL theme={null} SELECT r2.name, sc.name, r2.user_name FROM sys.service_classes AS sc, ( SELECT g.name, g.service_class_id, r.user_name FROM sys.groups AS g, ( SELECT u.user_name, ug.group_id FROM sys.users AS u, sys.user_groups AS ug WHERE u.id = ug.user_id AND u.user_name = 'jmack@test' ) AS r ) AS r2; ``` *Output* ```none Text theme={null} | group | service_class | user_name | |---------|-----------------|-------------| | analyst | Default | jmack@test | | analyst | high_priority | jmack@test | ``` ## Data Pipelines ### Pipeline Events While your pipeline is running, the pipeline generates events in the `sys.pipeline_events` system catalog table to mark significant checkpoints when something has occurred. In this example, you execute the `CREATE PIPELINE` and `START PIPELINE` SQL statements on the pipeline named `my_pipeline` and let the pipeline complete. As expected, the `CREATED`, `STARTED`, and `COMPLETED` events appear. For details about the lists of files, see the `sys.pipeline_files` system catalog table. Pipeline events include messages from many different tasks. These events are the background processes that execute across different Loader Nodes during pipeline operation. For details about tasks, you can query the `sys.tasks` and `sys.subtasks` system catalog tables. ```sql SQL theme={null} SELECT * FROM sys.pipeline_events ORDER BY event_timestamp; ``` *Output* ```sql SQL theme={null} +--------------------------------------+--------------------------------------+--------------------------------------+------------------------+-----------------------------------------------------------------------+----------------------------+ | pipeline_id | task_id | user_id | event_type | event_message | event_timestamp | |--------------------------------------+--------------------------------------+--------------------------------------+------------------------+-----------------------------------------------------------------------+----------------------------| | d8848467-9262-4bd6-84a7-1380c95f8b8b | | dde90d1b-bfcf-4b48-a251-b9ea812a7b26 | CREATED | Created pipeline my_pipeline | 2024-02-07 18:04:32.822690 | | d8848467-9262-4bd6-84a7-1380c95f8b8b | | dde90d1b-bfcf-4b48-a251-b9ea812a7b26 | STARTED | Started processing pipeline my_pipeline | 2024-02-07 18:39:52.682276 | | d8848467-9262-4bd6-84a7-1380c95f8b8b | | | FILE_LISTING_STARTED | File listing started for pipeline my_pipeline. This may take a while. | 2024-02-07 18:39:52.848000 | | d8848467-9262-4bd6-84a7-1380c95f8b8b | | | FILE_LISTING_COMPLETED | File listing completed for pipeline my_pipeline. 1 files were listed. | 2024-02-07 18:39:52.863000 | | d8848467-9262-4bd6-84a7-1380c95f8b8b | 61007c85-6744-42e7-9401-66e1a594e990 | | EXTRACTION_STARTED | Extraction started | 2024-02-07 18:39:58.367000 | | d8848467-9262-4bd6-84a7-1380c95f8b8b | 61007c85-6744-42e7-9401-66e1a594e990 | | EXTRACTION_COMPLETED | Extraction completed | 2024-02-07 18:39:59.417000 | | d8848467-9262-4bd6-84a7-1380c95f8b8b | 49c17f79-0108-4b05-aae5-9ae9a13abaca | | COMPLETED | Completed processing pipeline my_pipeline | 2024-02-07 18:40:03.808543 | +--------------------------------------+--------------------------------------+--------------------------------------+------------------------+-----------------------------------------------------------------------+----------------------------+ Fetched 7 rows ``` ### Pipeline Metrics Find metrics that measure the performance and activity of a pipeline in the `sys.pipeline_metrics` system catalog table. This table contains one row for each metric at a specified point in time. A common use of this data is to plot metric values over time to examine the behavior of a pipeline. **Metric Types** Metrics are either instantaneous or incremental: * Instantaneous metrics reflect the current value of a counter at the time of the metric collection. These values can increase or decrease based on the current state or rate of the metric. * Incremental metrics increment over time and reflect the cumulative value for a specified counter. **Example** The first snapshot of a metric appears when it is initialized. Afterward, the metric is updated every 10 seconds. The `updated_at` column shows the metric collection time. For example, if you are interested in the number of transformed record bytes, you can execute this query. The value increases over time as the pipeline runs. ```sql SQL theme={null} SELECT * FROM sys.pipeline_metrics WHERE name = 'count.record.bytes.transformed' ORDER BY updated_at; ``` *Output* ```sql SQL theme={null} +--------------------------------------+--------------------------------------+----------------+--------------+--------------------------------+-------------+----------------------------+ | pipeline_id | extractor_task_id | partition_id | sink_index | name | value | updated_at | |--------------------------------------+--------------------------------------+----------------+--------------+--------------------------------+-------------+----------------------------| | 0f51f1c0-f251-4407-8393-0f9bcbf18f28 | 034cf49d-364b-42b5-ad0a-37b7c237f16e | | | count.record.bytes.transformed | 7774214710 | 2024-02-07 22:35:27.546000 | | 0f51f1c0-f251-4407-8393-0f9bcbf18f28 | 034cf49d-364b-42b5-ad0a-37b7c237f16e | | | count.record.bytes.transformed | 16529202619 | 2024-02-07 22:35:37.546000 | | 0f51f1c0-f251-4407-8393-0f9bcbf18f28 | 034cf49d-364b-42b5-ad0a-37b7c237f16e | | | count.record.bytes.transformed | 25077088214 | 2024-02-07 22:35:47.546000 | | 0f51f1c0-f251-4407-8393-0f9bcbf18f28 | 034cf49d-364b-42b5-ad0a-37b7c237f16e | | | count.record.bytes.transformed | 33425410618 | 2024-02-07 22:35:57.546000 | | 0f51f1c0-f251-4407-8393-0f9bcbf18f28 | 034cf49d-364b-42b5-ad0a-37b7c237f16e | | | count.record.bytes.transformed | 41818859671 | 2024-02-07 22:36:07.546000 | +--------------------------------------+--------------------------------------+----------------+--------------+--------------------------------+-------------+----------------------------+ Fetched 5 rows ``` **Metric Scope** Each metric has a scope that defines its uniqueness. Be careful not to aggregate metrics across different scopes. The scope columns are `pipeline_id`, `extractor_task_id`, `partition_id`, and `sink_index`. If the value of any of these columns is NULL, the scope applies to all values in that dimension. For example, because the `partition_id` and `sink_index` columns are NULL for the metric `count.record.bytes.transformed`, its scope applies to the pipeline indicated by the `pipeline_id` and `extractor_task_id` across all sinks and partitions. An `extractor_task_id` is the internal identifier used when the Ocient System executes a pipeline process on a Loader Node. A pipeline might have many of these processes across time and loaders. **Example** Retrieve the most recent snapshot for each metric and aggregate the values to produce a single value for a pipeline. ```sql SQL theme={null} SELECT pipeline_id, name, SUM(most_recent_value) AS total_value FROM ( SELECT DISTINCT * FROM ( SELECT pipeline_id, extractor_task_id, partition_id, name, first_value (value) OVER (PARTITION BY pipeline_id, extractor_task_id, partition_id, name ORDER BY updated_at DESC) AS most_recent_value FROM sys.pipeline_metrics WHERE name LIKE 'count%' ) ) GROUP BY pipeline_id, name; ``` *Output* ```sql SQL theme={null} +--------------------------------------+--------------------------------+---------------+ | pipeline_id | name | total_value | |--------------------------------------+--------------------------------+---------------| | 0c7b32f1-37ca-43f7-96b6-69c266c978a5 | count.file.total | 64 | | 0c7b32f1-37ca-43f7-96b6-69c266c978a5 | count.record.bytes.transformed | 33644961 | | 0c7b32f1-37ca-43f7-96b6-69c266c978a5 | count.file.processed | 64 | | 0c7b32f1-37ca-43f7-96b6-69c266c978a5 | count.record.sent | 100000 | | 0c7b32f1-37ca-43f7-96b6-69c266c978a5 | count.record.durable | 100000 | +--------------------------------------+--------------------------------+---------------+ Fetched 5 rows ``` This output contains a single row for each metric for each pipeline. If you run a pipeline with 64 files and 100,000 records, the output might look like this. This table explains each column in the output. ## Related Links [System Catalog](/system-catalog) [Data Integrity and Storage](/data-integrity-and-storage) [Query Analysis](/query-analysis) [Data Definition Language (DDL) Statement Reference](/data-definition-language-ddl-statement-reference) [Cluster and Node Management](/cluster-and-node-management) [Users, Groups, and Service Classes](/users-groups-and-service-classes) # Distributed Tasks Source: https://docs.ocient.com/distributed-tasks Overview of distributed tasks in Ocient, including rebuild tasks and data integrity checks that run in the background to maintain system health and performance. You can manage distributed tasks in the System. These tasks are processes that execute in the background to keep the system running smoothly and efficiently. These DDL commands create tasks to rebuild damaged or missing segment data and check data integrity. For information on managing tasks, see [Manage Distributed Tasks](/manage-distributed-tasks). ## CREATE TASK Launches a new task. **Privileges** To create any task, you must have the System Administrator role, which has the `UPDATE` privileges on the system. **Syntax** ```sql SQL theme={null} CREATE TASK [ task_name ] TYPE { rebuild | check_disk | rebalance } [ LOCATION { SYSTEM | CLUSTER cluster_name | NODE node_name } ] [ OPTIONS task_option_map [, ... ] ] ``` To create a task, you must specify a task type `TYPE`: * [rebuild](#rebuild-task) — Reconstruct data segments by using erasure-coded data in the same segment group. * [check\_disk](#check_disk-task) — Verify segments on a node by computing checksums. * [rebalance](#rebalance-task) — Distribute data evenly across disks in the system. You can also optionally specify a location using the `LOCATION` keyword to execute the task. Supported values include: * `SYSTEM` — Targets all segments in the system. If you do not specify the `LOCATION` keyword, `SYSTEM` is the default value. * `CLUSTER` — Targets a specific cluster. * `NODE` — Targets a specific node. | **Parameter** | **Data Type** | **Description** | | ----------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `task_name` | String | Optional.

The identifier for the task. | | `TYPE` | String | Required. Specifies the type of the distributed task to create. Supported tasks are:

`rebuild` — Rebuild task
`check_disk` — Check disk task
`rebalance` — Rebalance task

Task names are case sensitive. | | `node_name` | String | This argument is required only if your task targets a node with the `LOCATION NODE` keywords.

The name of a specific node for the task to target. | | `cluster_name` | String | This argument is required only if your task targets a cluster with the `LOCATION CLUSTER` keywords.

The name of a specific cluster for the task to target. | | `task_option_map` | Key-value pairs | Optional. A comma-separated list of task-specific options in key-value pair format `key = value`. | ### `rebuild` Task Performing a rebuild segment operation reconstructs data segments by using erasure-coded data in the same segment group. This task allows the system to restore segments that are in the damaged `DAMAGED` or missing `MISSING` status to full performance. For details on rebuild tasks, see [Guide to Rebuilding Segments](/guide-to-rebuilding-segments). The `rebuild` task supports only the `SYSTEM` and `CLUSTER` locations. If you attempt the rebuild task on the node, the system returns an error. To rebuild a single node, use the cluster location with the `LOCATION CLUSTER` keywords. Include the name of the node you want to rebuild as one of the key-value options. `rebuild` **Options (**`task_option_map`**)** The rebuild task supports these options. You must specify these options as a comma-separated list of key-value pairs. | **Option Key** | **Value Type** | **Description** | | ---------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `node` | String | Limits the rebuild operation to only segments on the specified node. | | `segment_groups` | List | Limits the rebuild operation to the specified segment group identifiers. Specify this option as a bracketed list, e.g., `[ id1, id2, ... ]`. | | `table_id` | String | Limits the rebuild to segment groups belonging to a specific table identifier. | **Examples** **Rebuild All Segments on the System** This example performs a system-wide rebuild task. ```sql SQL theme={null} CREATE TASK TYPE rebuild LOCATION SYSTEM; ``` **Rebuild Segments on the Specific Table** This example rebuilds all segments on a system for the specified table with the table identifier `529a2e9d-d06c-46cd-a93b-624d3bed1c08`. ```sql SQL theme={null} CREATE TASK TYPE rebuild OPTIONS table_id = '529a2e9d-d06c-46cd-a93b-624d3bed1c08'; ``` **Rebuild Segments on the Specific Cluster** This example rebuilds segments on the Foundation cluster `my_lts_cluster1`. ```sql SQL theme={null} CREATE TASK TYPE rebuild LOCATION CLUSTER my_lts_cluster1; ``` **Rebuild Segments on the Specific Node** This example rebuilds segments on the Foundation Node `my_node1` in the Foundation Cluster `my_lts_cluster1`. ```sql SQL theme={null} CREATE TASK TYPE rebuild LOCATION CLUSTER my_lts_cluster1 OPTIONS node = 'my_node1'; ``` **Rebuild the Specific Segment Group** This example rebuilds segments in the segment group with the identifier `53` in the Foundation Cluster `my_lts_cluster1`. ```sql SQL theme={null} CREATE TASK TYPE rebuild LOCATION CLUSTER my_lts_cluster1 OPTIONS segment_groups = [ 53 ]; ``` **Create a Named Task** This example creates the rebuild task named `my_rebuild_task` in the Foundation Cluster `my_lts_cluster1`. ```sql SQL theme={null} CREATE TASK my_rebuild_task TYPE rebuild LOCATION CLUSTER my_lts_cluster1; ``` ### `check_disk` Task The `check_disk` task verifies the integrity of data segments on storage nodes by computing checksums. The task compares stored checksums with recomputed ones to detect disk corruption. `check_disk` **Options (**`task_option_map`**)** The `check_disk` task supports these options. You must specify these options as a comma-separated list of key-value pairs. | **Option Key** | **Value Type** | **Description** | | ---------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `fix` | Boolean | If you set this option value to `true`, the task modifies the storage cluster state based on the checksum result. For example, if the task finds a segment that fails the checksum, the task sets its segment state to `DAMAGED`.

Otherwise, the task reports any segments that fail the checksum, but it does not alter segment states.

The default value is `false`. | | `sample` | Boolean | If you set this option value to `true`, the task checks only a sample of data in each segment.

The default value is `false`. | | `only_unhealthy` | Boolean | If you set this option value to `true`, the task checks only segments that are not in the `INTACT` state.

The default value is `false`. | | `storage_ids` | List | This option limits the `check_disk` task to only a specific list of storage identifiers. Specify these storage identifiers as a bracketed list, e.g., `[ 'id1', 'id2', ... ]`. | **Examples** **Perform Checksum on the Specific Node** This example computes the checksum on all segments on storage node `my_node1`. ```sql SQL theme={null} CREATE TASK TYPE check_disk LOCATION NODE 'my_node1'; ``` **Perform a Checksum on the Specific Cluster** This example computes a checksum for a sample of blocks in each segment of the storage cluster `my_lts_cluster1` using the `sample = true` key-value pair. ```sql SQL theme={null} CREATE TASK TYPE check_disk LOCATION CLUSTER my_lts_cluster1 OPTIONS sample = true; ``` **Compute Checksum on the Specific Storage Identifier** This example computes a checksum for the segment with the storage identifier `529a2e9d-d06c-46cd-a93b-624d3bed1c08` on the storage node `my_node1`. ```sql SQL theme={null} CREATE TASK TYPE check_disk LOCATION NODE 'my_node1' OPTIONS storage_ids = [ '529a2e9d-d06c-46cd-a93b-624d3bed1c08' ]; ``` ### `rebalance` Task Rebalancing evenly distributes data across disks in the system. Execute this task if data is not properly balanced across the system, such as after adding nodes or drives. For details on performing a `rebalance` task, see [Expand and Rebalance System](/expand-and-rebalance-system). The `rebalance` task supports only `SYSTEM` and `CLUSTER` locations. If you attempt the rebalance task on a node, the system returns an error. **Example** This example performs a rebalance task across all foundation clusters. ```sql SQL theme={null} CREATE TASK TYPE rebalance; ``` ## CANCEL TASK Cancels a running task. **Privileges** To cancel a task, you must be assigned the System Administrator role. **Syntax** ```sql SQL theme={null} CANCEL TASK ('task_name' | 'task_id') ``` | **Parameter** | **Data Type** | **Description** | | ------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `task_name` | string | The name specified on task creation. This applies only if an optional name was created for the task. | | `task_id` | string | The unique identifier for the task in the system tables.
For rebuild tasks, you can find the unique identifier in the [sys.result\_cache](/system-catalog#sys-result_cache) table.
For check disk tasks, you can find the unique identifier in the [sys.subtasks](/system-catalog#sys-subtasks) table. | **Example** Cancel a task named `my_task`. ```sql SQL theme={null} CANCEL TASK 'my_task'; ``` ## DROP TASK Drops orphaned or completed tasks from the system. The task identifier must be a task Universally Unique IDentifier (UUID). The statement does not accept task names because names are not globally unique. A name becomes reusable after its prior task reaches a terminal status, which makes name-based lookup ambiguous. The terminal task statuses are `FAILED`, `CANCELLED`, `INVALID`, `COMPLETE`, and `QUIESCED`. By default, when you do not specify any optional keywords, the `DROP TASK` SQL statement drops a single leaf task. The database throws an error if the task has children or is not in a terminal status. **Privileges** To drop a task, you must be assigned the System Administrator role. If you have the System Administrator role, you can drop any task. If you are not an administrator, you can drop COALESCE tasks for which you have `CREATE` privileges or pipeline tasks for which you have `EXECUTE` privileges. **Syntax** ```sql SQL theme={null} DROP TASK [ IF EXISTS ] uuid [ CASCADE ] [ FORCE ] ``` | **Parameter** | **Data Type** | **Description** | | ------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------- | | `uuid` | string | The task UUID, which is the unique identifier for the task in the system tables. The statement does not accept task names. | The optional keywords are: * `IF EXISTS` — Prevents the statement from returning an error when the task is not found. * `CASCADE` — Drops the entire subtree rooted at the specified task. All tasks in the subtree must be in a terminal status. * `FORCE` — Skips the terminal-status checks. You can combine the `FORCE` with `CASCADE` keywords to force the drop of an entire subtree regardless of task status. Generally, terminal tasks automatically roll off the system, so executing the `DROP TASK` SQL statement is not necessary. However, the `FORCE` and `CASCADE` keywords can help you drop orphaned tasks from the system in the event of an internal lockup. Be aware that this statement can unintentionally delete the state of in-progress tasks. **Examples** **Drop a Single Leaf Task** This example drops a single leaf task using its UUID. ```sql SQL theme={null} DROP TASK '529a2e9d-d06c-46cd-a93b-624d3bed1c08'; ``` **Drop a Task Only If It Exists** This example drops a task and does not return an error if the system cannot find it. ```sql SQL theme={null} DROP TASK IF EXISTS '529a2e9d-d06c-46cd-a93b-624d3bed1c08'; ``` **Drop a Task and Its Subtree** This example drops the specified task and all tasks in its subtree. All tasks in the subtree must be in a terminal status. ```sql SQL theme={null} DROP TASK '529a2e9d-d06c-46cd-a93b-624d3bed1c08' CASCADE; ``` **Force the Drop of a Task Subtree** This example drops the specified task and its entire subtree, regardless of the task status. ```sql SQL theme={null} DROP TASK '529a2e9d-d06c-46cd-a93b-624d3bed1c08' CASCADE FORCE; ``` ## Related Links [System Catalog](/system-catalog) [Guide to Rebuilding Segments](/guide-to-rebuilding-segments) # Download Source: https://docs.ocient.com/download Redirect to the Ocient documentation landing page where you can download PDFs, browse release-specific docs, and access additional product resources. # Ensemble Models Source: https://docs.ocient.com/ensemble-models Learn how OcientML ensemble models (Bagging, Boosting, Stacking) combine base models to reduce errors and improve predictions, with SQL syntax and examples. Ensemble models improve predictive performance by combining the strengths of multiple individual machine learning models into a single, robust system. Instead of relying on one algorithm, ensemble techniques train a collection of base models and aggregate their predictions to reduce errors, lower variance, and prevent overfitting. ## Bagging Model Type: `BAGGING` A bagging model trains multiple base models in parallel on random subsets of the training data and aggregates their predictions. Use the bagging model to reduce variance and prevent overfitting. The bagging model utilizes child models embedded as options in JSON format (see `baseModels`). The bagging model trains all child models independently in parallel. When aggregating all the child model predictions, the bagging model uses majority voting for classification and averaging for regression. A bagging model can use any model type that supports classification or regression. ### Model Options #### Required `baseModels` — A JSON array defining the child models to train. Each object in the array specifies a model type, a count (how many of this model to create), and any options specific to the individual child model. `baseModels` **JSON Arguments** | **Argument** | **Data Type** | **Description** | | ------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | string | The model type (e.g., `"DECISION TREE"`, `"MULTIPLE LINEAR REGRESSION"`).

The selected model type should comply with the selected `taskType` (i.e., select a classification model if your `taskType` is `CLASSIFICATION`. | | `count` | integer | The number of instances of this model type to train. | | `options` | JSON object | JSON object containing options specific to that child model type, exactly as if you were creating that model directly. For a list of supported options, see the descriptions of model types. | `taskType` — Specifies the task type with values: `'CLASSIFICATION'` or `'REGRESSION'`. #### Optional `maxThreads` — An optional integer value that controls the maximum number of child models that the bagging model trains in parallel. This option does not affect the internal threading of each child. If supported, the `maxThreads` value of each child model controls the internal threading. The default value is 16. `bootstrap` — An optional Boolean value. If you set this option to `true`, the model uses bootstrap sampling with replacement. Each child trains on a random sample where rows can be repeated. Requires you to set the `noSnapshot` option to `false`. If you set this option to `false`, the model uses sampling without replacement (each row appears at most one time per child). The default value is `false`. `rowsPerChild` — An optional integer specifying the exact number of rows to use for each child model. `0` means use all available rows (only valid if you do not set the `fractionSelected` option). The default value is `0`. `fractionSelected` — An optional double value between 0.0 and 1.0 that specifies the proportion of rows to use for each child model. This option is unusable if you set the `rowsPerChild` option to greater than 0. The default value is `1.0` (use all rows). `inputsPerChild` — An optional integer that specifies the number of features (columns) to use for each child model. The default value is `CEIL(total_features / 3)`. `metrics` — An optional Boolean value. If you set this option to `true`, the model calculates and stores quality metrics on the training data after training completes. The default value is `false`. For classification, the quality metrics include: * Accuracy: The percentage of training rows where the model predicted the correct class. * Area Under the Curve (AUC): A score from 0 to 1 that measures how well the model distinguishes between two different categories. For regression, the quality metrics include: * Root Mean Squared Error (RMSE): The square root of the average squared difference between the predicted and actual values. * Adjusted R²\*\*:\*\* The coefficient of determination, adjusted for the number of predictors (features) in the model. This metric indicates how well the independent variables explain the variance in the dependent variable. The default value is `false`. `featureArray` — An optional Boolean value. If you set this option to `true`, the model expects only one array-type column as input, rather than multiple columns of training data. Each array row in the input column must be the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. ### Execute the Model This example creates a bagging model that, in effect, functions as a random forest by training 50 decision trees for a classification task. ```sql SQL theme={null} CREATE MLMODEL my_schema.bagging_classifier TYPE BAGGING OPTIONS ( 'taskType' = 'CLASSIFICATION', 'baseModels' = '[ { "type": "DECISION TREE", "count": 50, "options": { "maxDepth": 8, "splitMetric": "gini_impurity" } } ]', 'bootstrap' = 'true', 'fractionSelected' = '0.8', 'maxThreads' = '16', 'metrics' = 'true' ) AS SELECT feature1, feature2, feature3, label FROM my_schema.training_data; ``` After the training completes, execute the bagging model. ```sql SQL theme={null} SELECT my_schema.bagging_classifier(feature1, feature2, feature3) FROM my_schema.scoring_data; ``` ## Boosting Model Type: `BOOSTING` A boosting model is an ensemble technique that trains multiple base models sequentially, where each new model attempts to correct the errors of the combined previous models. Boosting reduces bias and errors in supervised learning tasks. The boosting model utilizes child models specified as options in JSON format (see the `baseModels` argument). The boosting model trains its child models one after another, with each child contributing to the final prediction, weighted by a set learning rate. A boosting model can use any model type that supports regression. ### Model Options #### Required `baseModels` — A JSON array describing the child models to train. Each object in the array specifies a model type, the number of instances of this model to create, and other options that depend on the model type. `baseModels` **JSON Arguments** | **Argument** | **Data Type** | **Description** | | ------------ | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | string | The OcientML model type (e.g., `"DECISION TREE"`, `"MULTIPLE LINEAR REGRESSION"`).

The selected model type should comply with the selected `taskType` (i.e., select a classification model if your `taskType` is `CLASSIFICATION`. | | `count` | integer | The number of instances of this model type to train sequentially. | | `options` | JSON object | JSON object containing options specific to that child model type, similar to creating that model directly. For a list of supported options of a child model, see the individual descriptions of model types. | `taskType` — Specifies the task type with values: `'CLASSIFICATION'` or `'REGRESSION'`. `learningRate` — A decimal value between 0.0 and 1.0 that tunes how much the model learns from each successive child. Lower values can lead to better generalization, but they require more trees (a higher `count` value in the `baseModels` JSON field). #### Optional `lossFunction` — A string value that determines how the model calculates prediction errors and what type of problem it solves. Accepted values are: * `'squared_error'` — Configures the model for regression tasks. Calculates errors as the squared difference between predicted and actual values. Use this value for predicting continuous numeric values (e.g., prices, temperatures, quantities). The target column must contain numeric values. The `lossFunction` option defaults to `squared_error` if you set the `taskType` option to `REGRESSION`. * `'log_loss'` — Configures the model for classification tasks. This model uses logistic loss to calculate prediction errors for probability-based predictions. The `lossFunction` option defaults to `log_loss` if you set the `taskType` option to `CLASSIFICATION`. `fractionSelected` — A double value specifying the proportion of rows (0.0 \< n \<= 1.0) to use for each child model. The default value is `1.0` (use all rows). `inputsPerChild` — An integer value specifying the number of features (columns) to use for each child model. When you set this option, the algorithm deterministically cycles through feature subsets. If you do not set this option, the model uses all available features for each child. `maxThreads` — An optional integer value that controls the maximum number of child models that the model trains in parallel. This option does not affect the internal threading of each child. If supported, the `maxThreads` value of each child model controls the internal threading. The default value is 16. `metrics` — An optional Boolean value. If you set this option to `true`, this option calculates and stores quality metrics on the training data after training completes. The default value is `false`. For classification, the quality metrics include: * Accuracy: The percentage of training rows where the model predicted the correct class. * Area Under the Curve (AUC): A score from 0 to 1 that measures how well the model distinguishes between two different categories. For regression, the quality metrics include: * Root Mean Squared Error (RMSE): The square root of the average squared difference between the predicted and actual values. * Adjusted R²\*\*:\*\* The coefficient of determination, adjusted for the number of predictors (features) in the model. This metric indicates how well the independent variables explain the variance in the dependent variable. `skipDropTable` — An optional Boolean value. If you set this option to `false`, the database deletes intermediate tables created during training. If you set this option to `true`, the database prevents deletion of intermediate tables (useful for debugging). The default value is `false`. `continuousFeatures` — An optional comma-separated list of feature indexes (1-based) that are continuous numeric variables (i.e., not categorical or discrete variables). `featureArray` — An optional Boolean value. If you set this option to `true`, the model expects only one array-type column as input, rather than multiple columns of training data. Each array row in the input column must be the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. ### Execute the Model This example creates a boosting model (similar to Gradient Boosted Trees but using generic boosting logic) that trains 100 decision trees sequentially for a regression task. ```sql SQL theme={null} CREATE MLMODEL my_schema.boosting_regressor TYPE BOOSTING OPTIONS ( 'taskType' = 'REGRESSION', 'baseModels' = '[ { "type": "DECISION TREE", "count": 100, "options": { "maxDepth": 4 } } ]', 'learningRate' = '0.1', 'lossFunction' = 'squared_error', 'metrics' = 'true' ) AS SELECT feature1, feature2, feature3, label FROM my_schema.training_data; ``` After training completes, execute the boosting model. ```sql SQL theme={null} SELECT my_schema.boosting_regressor(feature1, feature2, feature3) FROM my_schema.scoring_data; ``` ## Stacking Model Type: `STACKING` The stacking model combines multiple base models into a single, higher‑accuracy ensemble. A stacking model can use any other machine learning model that supports regression, classification, or clustering. A stacking model can produce better accuracy than a single model by incorporating the strengths of different models and feature subsets. Stacking trains two model levels: * Level‑zero models: one or more base models that train on the original features. * Level‑one model (meta‑model): a single model that trains on the predictions of the level‑zero models (plus any preserved features), and learns how to weight and combine them. A stacking model cannot use the Vector Autoregression, Feedforward Neural Network, or Association Rules models as level-zero or level-one models. ### Model Options #### Required The stacking model utilizes other models embedded as options in JSON format (see `levelZeroModels` and `levelOneModel`). Both are required. `levelZeroModels` — An array of JSON objects representing one or more base models. Internally, the system maps these models using the same strings as other machine learning model types. Any stacked models have the same features, requirements, and options as they would without stacking. `levelOneModel` — A JSON object that defines the single meta‑model that sits on top of all level‑zero models in a stacking ensemble. The object specifies the model type, name, and any model‑specific options for this level‑one model, which is trained on the outputs of the level‑zero models (and any preserved input columns) to produce the final prediction. JSON strings for both `levelZeroModels` and `levelOneModel` share these arguments unless noted otherwise in the description. | **Argument** | **Data Type** | **Description** | | -------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | string | The machine learning model type to use for this child (for example, `"RANDOM FOREST"`, `"LOGISTIC REGRESSION"`, `"KMEANS"`). Must be a valid OcientML model type that supports regression, classification, or clustering. | | `name` | string | Optional. A name for the child model. If you omit this argument, the system auto‑generates a name by attaching a suffix to the parent model name. | | `options` | JSON object | JSON object containing options specific to the chosen model type, exactly as if you were creating that model directly. For a list of options, see the individual descriptions of individual model types. | | `ignoreColumns` | string | Optional. Used only for `levelZeroModels`. The value is a comma-separated list of 1-based column indices to exclude from this level‑0 model training input. If you omit this argument, the model sees all available features, labels, and extra columns. | | `extraCallArguments` | string | Advanced optional argument. An array of argument lists. Each element is an array of strings representing extra literal arguments appended when the system executes this child model in the level‑one training SQL.

Level-zero child models can use any number of argument lists. Level‑one child models can use only one argument list. | #### Optional `maxThreads` — An optional integer value that controls the maximum number of level‑0 child models that the stacking model trains in parallel. This option does not affect the internal threading of each child. If supported, the `maxThreads` value of each child model controls the internal threading. A higher `maxThreads` value can reduce total training time for many level‑0 models, but increases concurrent resource usage on the cluster. The default value is 16. `noSnapshot` — An optional Boolean value that controls whether the stacking trainer first materializes the training query into an intermediate snapshot table. When you set the `noSnapshot` option to `false`, the model creates a temporary snapshot table from the input SQL and trains all level‑0 and level‑1 models against that fixed data set, ensuring consistent data even if the source changes. When you set the `noSnapshot` option to `true`, the trainer reads the input query directly without creating a snapshot (which can be faster but requires the underlying data not to change). If the query is a common table expression (i.e., it starts with a `WITH` clause), then stacking automatically forces the `noSnapshot` option value to `false` and logs a warning. The default value is `false`. `featureArray` — An optional Boolean value. If you set this option to `true`, the model expects only one array-type column as input, rather than multiple columns of training data. Each array row in the input column must be the same size. Regardless of whether you use this option, the model treats training data the same with labeling and weight scoring. The default value is `false`. `hasLabelColumn` — Specifies whether the training data includes a label column (necessary for supervised tasks like classification or regression). If you set this option to `true`, the input query must return features followed by a label column. If you set this option to `false`, the model assumes the input contains only features (typically for unsupervised tasks, such as clustering or dimension reduction models). The default value is `true`. `extraColumnCount` — A non-negative integer representing the number of columns present after the label column (e.g., for weights). Level-zero models do not use these columns as standard features, but you can pass these columns to the level-one model using the `extraCallArguments` argument logic. The default value is `0`. `preservedColumnsForLevelOne` — A list of integers representing column indices (1-based) from the input data. The stacking model passes these specified columns to the level-one model. This option is useful if the level-one model needs to access raw features to improve the base model predictions. If you do not specify this option, the default value is an empty list (i.e., no original features are preserved). ### Execute the Model This example demonstrates how a stacking model can embed level-zero and level-one child models. The JSON `OPTIONS` string includes two level-zero models (random forest and gradient boosted trees) and produces the final prediction using the level-one model (logistic regression). Each models specifies its own options for how to run. In the example, the random forest model (`rf_base`) uses 200 decision trees. This model explicitly ignores columns 3 and 4, training only on columns 1 and 2. In contrast, the gradient boosted trees model (`gbt_base`**)** uses 300 decision trees and all available columns. The level-one model uses logistic regression to combine the outputs of the two base models. The model takes the predictions from `rf_base` and `gbt_base` as its inputs and produces the final classification result. When you run the stacking model, the stacking model first executes the base models in parallel (up to the `maxThreads` value). The resulting predictions then go into the logistic regression meta-model to generate the final score. ```sql SQL theme={null} CREATE MLMODEL my_schema.stacking_classifier TYPE STACKING OPTIONS ( -- Level-0 (Base) Models 'levelZeroModels' = '[ { "type": "RANDOM FOREST", "name": "rf_base", "options": { "numTrees": 200, "maxDepth": 12 }, "ignoreColumns": "3,4" }, { "type": "GRADIENT BOOSTED TREES", "name": "gbt_base", "options": { "numChildren": 300, "learningRate": 0.05 } } ]', -- Level-1 (Meta) Model 'levelOneModel' = '{ "type": "LOGISTIC REGRESSION", "name": "stacking_meta", "options": { "maxIterations": 200, "regularization": 0.1 } }', -- Top-level STACKING options 'maxThreads' = '16', 'hasLabelColumn' = 'true' ) AS SELECT feature1, feature2, feature3, feature4, label FROM my_schema.training_data; ``` After training completes, execute the stacking model. ```sql SQL theme={null} SELECT my_schema.stacking_classifier(feature1, feature2, feature3, feature4) FROM my_schema.scoring_data; ``` ### Related Links [Regression Models](/regression-models) [Classification Models](/classification-models) [Clustering and Dimension Reduction Models](/clustering-and-dimension-reduction-models) # Error Tolerance in Data Pipelines Source: https://docs.ocient.com/error-tolerance-in-data-pipelines Configure the ERROR LIMIT clause on Ocient data pipelines to set how many record-level errors a batch tolerates before the pipeline enters a FAILED status. ## Error Limits The error limit for SQL statements such as `START PIPELINE my_pipeline ERROR LIMIT 100` defines the number of record-level errors that a batch of files tolerates before failing the entire pipeline. The error process is: * Periodically, the System assigns a batch of pending files to a Loader Node. * The system enforces the maximum error limit for each batch. * Files that have at least one record-level error reach the terminal status `LOADED_WITH_ERRORS` after all processing is complete. * If the file batch reaches the error limit, the current file reaches the status `FAILED`, and then the pipeline status becomes `FAILED`. Processing stops on the pipeline. By default, `BATCH` pipelines run with `ERROR LIMIT 0`. This permits zero errors on a pipeline. This setting stops the pipeline on the first error and stores the error in `sys.pipeline_errors.` When you restart the pipeline, it retries the failed file and failed record. The system does not duplicate any previously loaded data. To complete the load, fix the issue with the data or the pipeline definition or increase the error limit when you restart the pipeline. ## Unrecoverable File Errors You can manage unrecoverable file errors such as Gzip decompression errors, tokenization errors, or missing files by starting the pipeline with the `FILE_ERROR` option. The strictest setting is `FILE_ERROR FAIL`. When you use the `FAIL` setting, any file-level error causes the pipeline to fail. If any file is missing or cannot be processed, the pipeline marks the file `FAILED`, sets the pipeline status to `FAILED`, and processing stops on the pipeline. The most tolerant setting is `FILE_ERROR TOLERATE`. Use this setting in a statement such as `START PIPELINE my_pipeline ERROR FILE_ERROR TOLERATE`. When you use the `TOLERATE` setting, missing files reach the terminal status `SKIPPED`. Files that encounter other file-level errors reach the terminal status `LOADED_WITH_ERRORS`. When the Ocient System encounters a file-level error, processing stops on the file and continues with the next file in the partition. When you use the `TOLERATE` setting, the pipeline automatically executes with an unlimited error limit. ## Recoverability If a pipeline fails when it tolerates no errors, manually fix the row or file where the error occurs and restart the pipeline so that the load proceeds with all records correctly deduplicated. Restarting the pipeline without fixing the error but with tolerance of record or file-level errors also allows the load to proceed with all records correctly deduplicated. However, restarts do not guarantee correct deduplication if you apply manual fixes after this point. When a pipeline fails, the number of loaded rows is nondeterministic. As a result, the Ocient System does not guarantee the reflection of any manual modification of files with the statuses `SKIPPED`, `LOADED_WITH_ERRORS`, or `FAILED` in a restart operation. However, if the pipeline fails due to a low error limit, you can raise the error limit on a restart operation to allow the pipeline to make further progress. You can use the `BAD_DATA_TARGET` option to capture failing records for troubleshooting and reloading. ## Restarts and Deduplication Ocient pipelines enable the restart of a pipeline without creating duplicate data in the target tables. However, there are some limitations for each type of data source. For details, see [Data Pipelines](/data-pipelines). ## Related Links [Data Pipelines DDL Reference](/data-pipelines) [START PIPELINE](/data-pipelines#start-pipeline) [Data Pipeline Load of JSON Data from Kafka](/data-pipeline-load-of-json-data-from-kafka) # Errors and Warnings Source: https://docs.ocient.com/errors-and-warnings Reference for handling errors and warnings in Ocient SQL queries, data pipelines, and system operations, including common codes and resolution guidance. The System produces these error and warning codes as part of its operation. Refer to these codes for their description. You can also execute this SQL statement using the `sys.sql_messages` system catalog table to view the codes and descriptions in the database: `SELECT * FROM sys.sql_messages`. ## Error Codes The error codes are negative values that split into ranges by number such as -100s, -200s, and so on. | **-1xx Codes** | **Description** | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | -100 | Unknown error occurred. | | -101 | This is an internal error that indicates that a client (driver or connector) received an invalid response from the server. | | -102 | Internal error that indicates a corruption of the serialized result set sent from the server to the client. | | -103 | Internal error that indicates an unexpected condition has happened. | | **-2xx Codes** | **Description** | | -200 | A client attempted a connection with a malformed connection string. | | -201 | The connection to the server failed. | | -202 | The handshake with the server failed. | | -203 | The connection to the server was unexpectedly closed. | | -204 | A network communications error has occurred. | | -205 | The client tried to connect to a database with a specified name, but no database with that name exists. | | -206 | The target node is offline. | | **-3xx Codes** | **Description** | | -300 | An argument to a driver method execution was invalid. | | -301 | A driver method executed, but the input argument object was already closed. | | -302 | The cursor was not positioned in a valid row. | | -303 | An attempt was made to create a new result set, but the previous result set for the connection was still open. | | -304 | The user and password combination was incorrect. | | -305 | A statement that was not a query was passed to executeQuery(). | | -306 | A statement that was a query was passed to executeUpdate(). | | -307 | A NULL output buffer was passed to a driver method, where a NULL buffer is not allowed. | | -308 | A driver execution was made that is not implemented. | | -313 | An invalid parameter marker number was specified. | | -316 | No database connection exists. | | -317 | Invalid data was detected for a parameter marker. | | -318 | The cursor was not in the required state. | | -319 | An invalid buffer length was specified. | | -320 | There is currently no result set. | | -321 | A NULL indicator pointer was required but not specified. | | -322 | A connection attribute that is not supported was specified in SQLBrowseConnect. The four supported attributes are: "DRIVER", "DSN", "UID", "PWD". | | -323 | An invalid descriptor record index was specified. Valid indices start at 1 and increase to the size specified by executing SQLSetDescField with the SQL\_DESC\_COUNT field. | | -324 | An invalid or inconsistent descriptor field type was specified in SQLSetDescRec. | | -325 | An invalid descriptor field was specified in SQLSetDescField. | | -326 | The target handle passed to SQLCopyDesc was an IRD. | | -327 | An implicit descriptor handle that was not the original was the specified value in the execution of SQLSetStmtAttr with the SQL\_ATTR\_APP\_PARAM\_DESC or SQL\_ATTR\_APP\_ROW\_DESC attribute. | | -328 | Divide by zero. | | -329 | Invalid floating point operation that resulted in NaN or Inf. | | -330 | The action conflicts with the current state of the database. | | -331 | The unconditional DELETE FROM statement is rewritten using the TRUNCATE statement. In this case, the database does not create a plan that would appear in the results from the execution of the EXPLAIN statement. | | -332 | Already in use. Used mostly in DDL when a resource or option is already assigned to the object. | | -333 | Optional value was substituted. | | -334 | This driver is unknown or the version support has been removed. | | -335 | The execution of a client method was invalid. | | -336 | Cannot execute the new SQL statement as the system is processing a prior statement using the same connection in the background. | | **-4xx Codes** | **Description** | | -400 | One of the referenced columns does not exist. | | -401 | The requested data type conversion was invalid. | | -402 | Unknown data type. | | -403 | A column name was ambiguous in the context where it was used. | | -404 | Two incompatible data types were specified as return types in a case statement. | | -405 | Join leads to a duplicate column name. | | -406 | Invalid POINT conversion from NULL. | | -407 | Column value is larger than internal limits. | | **-5xx Codes** | **Description** | | -500 | Generic syntax error. | | -501 | The SQL statement is not valid or not supported in the context where it was used. | | -502 | A duplicate common table expression name was detected. | | -503 | The number of columns does not match the SELECT statement. | | -504 | A negative limit was specified. | | -505 | A negative offset was specified. | | -506 | An incorrect number of arguments was specified in a function. | | -507 | Aggregation of a constant value is invalid. | | -508 | An argument to a function had an incorrect data type. | | -509 | An invalid comparison was detected. | | -510 | Aggregation was used in a context where it is not allowed (possibly something in a WHERE clause that belongs in a HAVING clause). | | -511 | A HAVING clause did not have any aggregation in it. | | -512 | An ORDER BY clause referenced a column by position and that position was out of range. | | -513 | An expression of type list cannot be returned in a result set. | | -514 | You cannot have an expression of type list within another list. | | -515 | A list expression contains incompatible types. | | -516 | The function does not exist. | | -517 | If the result of an expression is a time interval type, the result must be cast to an integral type. | | -518 | An invalid matrix literal was specified. | | -519 | A matrix literal had incorrect dimensions. | | -520 | An invalid query priority was specified. | | -521 | A column reference in a SELECT statement is invalid due to aggregation. | | -522 | Columns for the first and second SELECT statements for the UNION, EXCEPT, or INTERSECT statement are not compatible. | | -523 | An expression in the CASE condition is not allowed. | | -524 | An expression in the JOIN condition is not allowed. | | -525 | Aggregation is not allowed in the PARTITION BY or ORDER BY clauses of a windowed aggregate. | | -526 | A non-constant value was found where a constant was expected in an argument of a windowed aggregate. | | -527 | Range-based framing does not allow the PRECEDING or FOLLOWING keywords. | | -528 | ORDER BY with a column ordinal is not allowed in a windowed aggregate. | | -529 | A frame specification is not allowed if no ORDER BY clause is specified in a windowed aggregate. | | -530 | UNBOUNDED PRECEDING is only allowed on the starting frame bound. | | -531 | UNBOUNDED FOLLOWING is only allowed on the ending frame bound. | | -532 | The ending frame bound cannot be less than the starting frame bound. | | -533 | No frame specification is allowed for this particular windowed aggregate. | | -534 | A window function requires a positive integer argument. | | -535 | The specified window function requires an ORDER BY statement in the OVER() clause. | | -536 | Invalid aggregation. | | -537 | Invalid expression. | | -538 | A CASE expression within a CASE expression is not allowed. | | -539 | Window function was used in a context where it is not allowed (perhaps in a predicate of a WHERE or HAVING clause). | | **-6xx Codes** | **Description** | | -600 | A referenced table does not exist. | | -601 | The specified database already exists. | | -602 | The specified table already exists. | | -603 | The specified view already exists. | | -604 | The specified view was not found. | | -605 | The specified database was not found. | | -606 | The specified storage space was not found. | | -607 | The specified storage space already exists. | | -608 | The specified user does not exist. | | -609 | The specified user already exists. | | -610 | The specified password for the new user is invalid. | | -611 | The specified group name already exists. | | -612 | The specified group does not exist. | | -613 | The specified connection already exists. | | -614 | The specified connection was not found. | | -615 | The referenced translation does not exist. | | -616 | A translation with that name exists. | | -617 | The specified user was not found in this group. | | -618 | The provided storage space has no storage cluster assigned. | | -619 | The referenced table is a system table and not allowed for the export operation. | | -620 | The referenced pipeline does not exist. | | -621 | A pipeline with that name exists. | | -622 | The pipeline is invalid. | | -623 | The pipeline is unable to be correctly compiled. | | -624 | The referenced task does not exist. | | -625 | An index with that type and on that column already exists. | | -626 | An index with that name already exists. | | -627 | The referenced table ID no longer exists. | | -628 | A connectivity pool with that name already exists. | | -629 | Cannot find a connectivity pool with the specified name. | | -630 | Cannot drop the last connectivity pool from a node. | | -638 | Could not find a SCHEMA with that name (Schema does not exist). | | **-7xx Codes** | **Description** | | -700 | The user does not have read authorization on a referenced table. | | -701 | The user does not have authorization to create a database. | | -702 | The user does not have modify authorization on a referenced table. | | -703 | The user does not have authorization to create a view. | | -704 | The user does not have modify authorization on a referenced database. | | -705 | The user does not have modify authorization on a referenced view. | | -706 | The user does not have authorization to drop a referenced table. | | -707 | The user does not have authorization to drop a referenced row. | | -708 | The user does not have authorization to drop a referenced view. | | -709 | The user does not have authorization to drop a referenced database. | | -710 | The user does not have authorization to create a table. | | -711 | The user does not have authorization to create a storage space. | | -712 | The user does not have authorization to drop a referenced storage space. | | -713 | The user does not have authorization to create a user. | | -714 | The user does not have authorization to drop a user. | | -715 | The user does not have the authority to create a group. | | -716 | The user does not have the authority to drop the group. | | -717 | The user does not have the authority to create a connection. | | -718 | The user does not have the authority to drop the connection. | | -719 | The user is not authorized to take this action. | | -720 | The user does not have the authority to modify the group. | | -721 | The user does not have the authority to execute a plan. | | -722 | The user does not have the authority to execute an inline plan. | | -723 | The user does not have the authority to create a connectivity pool. | | -724 | The user does not have the authority to drop the connectivity pool. | | -725 | The user does not have the authority to modify the connectivity pool. | | -726 | Unable to drop some objects in the request. | | -728 | The user does not have the authority to drop the SCHEMA (Objects exist in the schema). | | -733 | Session has expired. Refresh is necessary. | | **-8xx Codes** | **Description** | | -800 | An MLMODEL with that name was not found. | | -801 | An MLMODEL with that name already exists. | | -802 | The user does not have the authority to create an MLMODEL. | | -803 | The user does not have the authority to drop the MLMODEL. | | -804 | Unable to create an MLMODEL due to a singular matrix. | | -805 | Unable to create an MLMODEL because the training set is empty. | | -806 | Unable to perform LUP decomposition on this singular matrix. | | -807 | The user does not have the authority to modify the MLMODEL. | | **-9xx Codes** | **Description** | | -900 | Operation was canceled or aborted. | | -901 | I/O error. | | -902 | Scalar Subquery Cardinality Violation. | | -903 | Numeric value out of range. | | -904 | Plan compilation error. | | -905 | TKT limit reached. | | -906 | Out of memory. | | -907 | Segment not available. | | -908 | System is still initializing. | | -909 | Out of temporary disk space. | | -910 | Operation was automatically killed due to drop in the client connection. | | -911 | Failed to acquire OSN (Ownership Sequence Number). Data for querying is no longer available to the storage cluster. | | **-10xx Codes** | **Description** | | -1000 | The privilege target already has the privilege. | | -1001 | The user does not have the authority to grant this privilege. | | -1002 | The privilege target does not have that privilege. | | -1003 | The referenced role does not exist. | | -1004 | The provided object ID was not found. | | -1005 | Create task failure. | | -1006 | Task already in progress. | | -1007 | Could not find the supplied service class. | | -1008 | Supplied service class definition is invalid. | | **-11xx Codes** | **Description** | | -1100 | No available service classes for the user have available query slots. | | -1101 | The specified node is not found. | | -1102 | Segment not found. | | -1111 | Cluster not found. | | -1112 | A cluster with that name exists. | | -1120 | Query exceeded service class time limit. | | -1121 | Query exceeded session time limit. | | -1122 | Query exceeded query time limit. | | -1123 | Query exceeded service class row limit. | | -1124 | Query exceeded session row limit. | | -1125 | Query exceeded query row limit. | | -1126 | Query exceeded service class temporary disk limit. | | -1127 | Query exceeded session temporary disk limit. | | -1128 | Query exceeded query temporary disk limit. | | -1129 | Too many columns in result set for the service class. | | -1130 | Too many columns in result set for the session. | | -1131 | Too many columns in result set for the query. | | **-12xx Codes** | **Description** | | -1200 | The security integration is disabled. | | -1201 | The security integration does not exist. | | -1202 | The request failed due to a network communications error. | | -1203 | The Single Sign On request identifier could not be found. | | -1204 | No session associated with the connection exists. | | -1205 | The supplied security token is invalid. | | -1206 | The supplied security token signature does not match the expected value. | | -1207 | The supplied security token has a signature fingerprint from an unknown entity. | | -1208 | The supplied security token has expired. | | -1220 | A network communications error occurred. | | -1221 | The OpenID provider rejected the request that indicates an internal server error. | | -1222 | The OpenID provider returned an invalid response that indicates an error with the provider. | | -1223 | The request\_uri in the Authorization Request returns an error or contains invalid data. | | -1224 | The OpenID provider does not support use of the request parameter. | | -1225 | The OpenID provider does not support use of the request\_uri parameter. | | -1226 | The OpenID provider discovery document is invalid. | | -1227 | The database OpenID configuration is invalid. | | -1228 | The OpenID ID or access token has expired. | | -1229 | The ID or access token was signed by a key not known to the OpenID provider. | | -1230 | The signing key returned by the OpenID provider is invalid. | | -1231 | The token contains an invalid claim. | | -1232 | The token was issued by an unknown OpenID provider. | | -1233 | The token was issued for an alternate audience. | | -1234 | The token signature is invalid, which implies the token was manipulated enroute to the database. | | -1235 | The specified ID token is invalid. | | -1236 | The specified Access token is invalid. | | -1237 | The refresh token returned from the provider is invalid. | | -1238 | The authentication server has timed out while waiting for client authentication. | | -1250 | The client is not authorized to request an authorization code using this method. | | -1251 | The resource owner or authorization server denied the request. | | -1252 | The authorization server does not support obtaining an authorization code using this method. | | -1253 | The requested scope is invalid, unknown, or malformed. | | -1254 | The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed. | | -1255 | The authorization server encountered an unexpected condition that prevented it from fulfilling the request. | | -1256 | The authorization server is currently unable to handle the request due to a temporary overloading or maintenance of the server. | | -1257 | Cannot lock table. | | -1258 | Cannot unlock table. | | -1259 | Cannot delete from a join or a view with multiple tables. | | **Other** | **Description** | | -9999 | Undefined exception. | ## Warning Codes The warning codes are positive values. | **Warning Code** | **Description** | | ---------------- | ---------------------------------------------------------------------- | | +300 | Data received by a JDBC driver was truncated. | | +301 | The driver is below the recommended threshold for the warning version. | | +900 | A query returned an empty result set. | | +901 | The query is estimated to have a long execution time. | | +1113 | The referenced object does not exist. | | +1114 | An object with that name exists. | | +1115 | Not authorized to take this action. | | +1116 | Restart the node for changes to take effect. | | +1117 | Complete system restart required for changes to take effect. | | +1118 | Statement contains invalid syntax. | | +1119 | Unable to drop some objects in the request. | ## Related Links [SQL Reference](/sql-reference) [Data Definition Language (DDL) Statement Reference](/data-definition-language-ddl-statement-reference) [Data Integrity and Storage](/data-integrity-and-storage) [Query Analysis](/query-analysis) [System Catalog](/system-catalog) # Exabyte Scalability Design Source: https://docs.ocient.com/exabyte-scalability-design How the Ocient System scales to exabyte-class data sets using its distributed compute-adjacent storage architecture, parity, and high-throughput design. ## Cluster of Clusters Various system components in have each been designed to scale not only with hardware capabilities as technology progresses, but also in number of nodes storing and processing data. Like any distributed system, issues of fault tolerance, availability, and communication all become increasingly challenging as the number of units increases. The Ocient scalability approach can be broadly described as deriving from ***clusters of clusters***: smaller, locally connected clusters that manage their own availability and local state independently and without consideration to any other clusters in the system. The design goal of this is to minimize global state and global synchronization in any aspect of query processing, fault management, data loading, or data ownership. There are different kinds of clusters: **metadata** clusters, **storage** clusters, **VM** clusters, and **loading** clusters. An Ocient deployment can contain zero of more of each. With respect to scalability, each cluster runs its own Raft-based consensus protocol and maintains its own local shared state with it. Individual nodes can belong to multiple clusters and participate in multiple disjoint consensus protocols at the same time, but system scalability derives from the fact that no node has any global view of system state, and indeed, no such global state exists. This allows Ocient to quantify the upper bound on the `N` parameter described in the erasure coding section. Recall that `N` is the number of segments in a segment group. In a **storage cluster**, segment ownership and availability is confined to and maintained in the cluster-local consensus protocol. A necessary implication of data management rules for atomic (but still local) state changes is that *a particular segment group must be stored entirely within one storage cluster.* And because the Ocient System cannot place two segments from the same group onto the same node, the value `N` cannot exceed the number of nodes in a target storage cluster. This finally means that the upper bound on `N` is the practical limit on the consensus protocol implementation, coupled with the design goal of not allowing clusters to be "too big": somewhere in the range of 20 - 40, depending on deployment and schema details. ## Query Trees (and Plan Trees) Query execution necessarily spans multiple VM and storage clusters. A key phase of query processing, prior to any actual computation is the ***query probe***, which establishes the ***query tree*** to be used for that particular query. This tree fixes, for the particular execution with which it is associated, the set of nodes that provide which segments across *all* storage clusters, as well as the set of nodes that co-participate in the distributed execution of the query. The former is important because, as nodes and drives come and go, and as new data is loaded and old data is removed, it is imperative that all segments that should be included in a query are actually included exactly one time. The latter is important because each node might need to know exactly which subset of peers in the network are performing which aspects of query execution. In both cases, the probe phase enables query execution to proceed without any global coordination or synchronization. This graphic shows a query execution tree, which can scale up by increasing the number of clusters (represented horizontally in the graphic). Example query execution tree that shows four levels of multiple clusters At the top level of the query execution tree is the root operator, which has zero or more additional plan levels below it, depending on the complexity of the query. Each circle represents individual nodes, which provide segments across the storage clusters or specific tasks for the execution of the query.    Vertically, the graphic shows different plan levels that fan out among the nodes for the stages of the query plan, actions such as transformations, sorting, grouping or other operations. The leaf level at the bottom of the graphic represents the I/O operation that reads the queried data from storage.   When the system initiates query execution, each cluster executes its operations in parallel. For the most part, clusters operate in isolation unless they need to communicate for operations such as a join or shuffle.  When referring to the query plan, each level is delineated by the presence of a special ***gather operator*** that represents the network boundary between levels. All the plan operators between root and the first gather operator are said to *execute at level 1.* Any plan operators that exist between the first gather operator and the second gather operator execute at level 2, and so on, until the final gather operator and the set of operators below it until the leaf I/O operators that represent the leaf level. Query tree operators for three levels When referring to the query tree, each gather operator represents a fan-out from one level N gather operator to some fixed number of N+1 nodes. During execution, a specific level N node is responsible for coordinating and amalgamating partial query results from the level N+1 nodes below it. By definition, there is exactly one level 1 node in a query, fanning out to some number of level 2 nodes, each of which itself fans out to some number of level 3 nodes, and so on. The fan-out factor is closely aligned with the cluster sizes in use. In both cases, the tree structure promotes scalable query execution in a fault-tolerant way. ## Related Links [Multiple Storage Clusters for Loading Data](/multiple-storage-clusters-for-loading-data) [Query Ocient](/query-ocient) ## Related Videos [At the Whiteboard with Ocient: Scale](https://www.youtube.com/watch?v=RpoTOmBKV8I) # Expand and Rebalance System Source: https://docs.ocient.com/expand-and-rebalance-system Expand an Ocient System by adding storage and SQL nodes, then rebalance segments across the cluster to maintain capacity, performance, and fault tolerance. If your system needs more resources, you can expand it by installing additional nodes and drives. The steps for expanding a system depend on the type of node you are adding. These sections explain the steps for adding each node type. Foundation Nodes include additional steps for adding more storage drives and rebalancing the system. For information on how to replace nodes that have failed or are damaged, see [Replace Nodes](/replace-nodes). For information on replacing drives that have failed or are damaged, see [Replace an NVMe Node Drive](/replace-an-nvme-node-drive). ## Prerequisites These prerequisites apply to adding any node type: * The process requires either the System Administrator role or the granted UPDATE privilege on SYSTEM. * Any new nodes should have the same OS kernel version as other nodes in the system. * Any new nodes should have the same Ocient System version as the other nodes. For details, see [Ocient Application Installation](/ocient-application-installation). * Ensure that your system meets the requirements outlined in [Ocient System Bootstrapping](/ocient-system-bootstrapping). * Before adding nodes, stop any query or loading process running on your system. ## Add Foundation Nodes A core component of an Ocient System, the Foundation Nodes store user data in Ocient and perform the bulk of query processing. By adding more Foundation Nodes, a system can support extra storage capacity as well as better performance for query processing. This process requires rebalancing the data to distribute the query processing across all available hardware, including newly installed disks or nodes.  ### System Considerations * No individual cluster should have more than two petabytes of storage. * Each cluster should have the same number of Foundation Nodes. ### Tutorial Create the `/var/opt/ocient/bootstrap.conf` file on the new node using this YAML example. This YAML file should follow a similar format and parameters as the `bootstrap.conf` files on your other nodes. Replace the `` with the IP or Hostname of a node running the Administrator Role on your system. If you are not using DNS, use `nodeAddress` instead of `adminHost`. Use a user account with System Administrator privileges for your system for the `adminUserName` and `adminPassword`. ```yaml YAML theme={null} adminHost: adminUserName: my_admin adminPassword: example_password ``` Start the new node by using this command. ```shell Shell theme={null} sudo systemctl start rolehostd ``` This process takes about a minute to complete. This step accepts the node into the system, but it has not been assigned a role or a storage cluster. To validate that the node is on the system, execute this query using the `sys.nodes` system catalog table. ```sql SQL theme={null} SELECT name, status FROM sys.nodes ORDER BY 1; ``` *Output* ```sql SQL theme={null} name status -------------------------------------------------------------------- admin01 ACCEPTED admin02 ACCEPTED admin03 ACCEPTED loader01 ACCEPTED loader02 ACCEPTED loader03 ACCEPTED foundation01 ACCEPTED foundation02 ACCEPTED foundation03 ACCEPTED foundation04 ACCEPTED foundation05 ACCEPTED foundation06 ACCEPTED foundation07 ACCEPTED foundation08 ACCEPTED foundation09 ACCEPTED foundation10 ACCEPTED foundation11 ACCEPTED foundation12 ACCEPTED foundation_new ACCEPTED sql01 ACCEPTED sql02 ACCEPTED ``` The output shows the new node, named `foundation_new` in this example, listed as `ACCEPTED` in the `status` column. Execute this `ALTER CLUSTER` SQL statement to add the new node to a storage cluster. In this example, replace `storage_cluster_1` and `foundation_new` with your cluster name and node name, respectively. ```sql SQL theme={null} ALTER CLUSTER "storage_cluster_1" ADD PARTICIPANTS "foundation_new"; ``` This example adds only one new node. For information on adding multiple nodes, see [ALTER CLUSTER ADD PARTICIPANTS](/cluster-and-node-management#alter-cluster). Restart the `rolehostd` process on all nodes in the system by running the following series of commands at the shell terminal. First, stop the `rolehostd` process on all nodes. ```shell Shell theme={null} sudo systemctl kill -s SIGKILL rolehostd ``` Confirm that the `rolehostd` process is no longer running. ```shell Shell theme={null} sudo systemctl status rolehostd ``` Start the `rolehostd` process on the system again. ```shell Shell theme={null} sudo systemctl start rolehostd ``` You can verify the new node is active by executing this SQL query after you connect to the database. ```sql SQL theme={null} SELECT n.name, ns.operational_status FROM sys.node_status ns JOIN sys.nodes n ON ns.node_id = n.id ORDER BY n.name; name operational_status ------------------------------------------------------------------------------------------ admin01 Active admin02 Active admin03 Active loader01 Active loader02 Active loader03 Active foundation01 Active foundation02 Active foundation03 Active foundation04 Active foundation05 Active foundation06 Active foundation07 Active foundation08 Active foundation09 Active foundation10 Active foundation11 Active foundation12 Active foundation_new Active sql01 Active sql02 Active ``` Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see [Rebalance System](#rebalance-system). ## Add More Drives to Foundation Nodes Foundation Nodes can support extra storage by adding additional NVMe drives. This tutorial shows the steps to integrate new drives into an existing Ocient System. ### System Considerations * For best performance, all Foundation Nodes should have equal storage capacity. * No individual cluster should have more than two petabytes of storage. ### Prerequisites * The process requires `systemctl` access on your system OS. * Any new NVMe drives added to the system must be blank and unpartitioned. ### Tutorial Shut down the `rolehostd` process from the shell prompt. ```shell Shell theme={null} sudo systemctl stop rolehostd ``` Install the new storage drives in your system. Restart the node from the shell prompt. ```shell Shell theme={null} sudo systemctl restart rolehostd ``` The `rolehostd` process recognizes the new drive. To confirm the new drives are active and running, you can connect to your system and query the [sys.storage\_device\_status](/system-catalog#sys-storage_device_status) system catalog table. To use this example query, replace `` with the name of the node where you are adding a drive. ```sql SQL theme={null} SELECT n.name AS node_name, s.node_id, s.id AS serial_number, s.pci_address, s.device_status, s.device_model FROM sys.nodes n JOIN sys.storage_device_status s ON n.id = s.node_id WHERE n.name = ''; ``` *Output* ```none Text theme={null} |node_name |node_id |serial_number |pci_address |device_status|device_model | |-----------|------------------------------------|------------------------------------|--------------------------------------------------------|-------------|---------------------------------------| |foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|6156962f-fcf6-4299-bbd7-2618fc6f1d00|/var/opt/ocient/6156962f-fcf6-4299-bbd7-2618fc6f1d00.dat|ACTIVE |PCIe Data Center SSD INTEL SSDPE2ME800G4| |foundation0|9330a0b3-b3b7-4503-949c-043b196c0cc4|e6a4b24f-0d49-4704-b7ce-18af163c0701|/var/opt/ocient/e6a4b24f-0d49-4704-b7ce-18af163c0701.dat|ACTIVE |PCIe Data Center SSD INTEL SSDPE2ME800G4| ``` This output lists the serial numbers of all drives in the `foundation0` node, including their statuses. Perform a rebalance task to distribute your data across your newly expanded system evenly. For details, see [Rebalance System](#rebalance-system). ## Rebalance System `REBALANCE` task execution redistributes your data evenly across your segment groups and clusters. Rebalance your system if you have recently installed new hardware, particularly new nodes or drives. ### System Considerations Currently, the `REBALANCE` task does not move damaged segment groups. If you suspect that you might have a significant number of damaged segment groups, you can execute these steps to check and fix the groups: 1. Ensure all nodes are online. 2. Check whether you have damaged segment groups using the `sys.segment_groups` system catalog table. If damaged segment groups are present, execute a `REBUILD` task to rebuild the groups before rebalancing the system. For details, see [Guide to Rebuilding Segments](/guide-to-rebuilding-segments). 3. Execute the `REBALANCE` task using this tutorial. You can find information in the `sys.degraded_segment_groups` system catalog table to identify the damaged segment groups that need fixing. This table shows all segment groups with the `DAMAGED` or `UNAVAILABLE` state. You can also check the `sys.storage_used` system catalog table, which shows approximately the same number of used bytes `used_bytes` for each node entry with the same table after the rebalance execution. ### Prerequisites * Only one rebalance task can execute at a time on the system. The system logs an error if you try to start a second rebalance task.  * You must have the System Administrator role, or be granted the UPDATE privilege on SYSTEM. ### Tutorial Execute a `SELECT` SQL statement to view how the system has distributed the existing data using the `sys.tables` and `sys.nodes` system catalog tables. ```sql SQL theme={null} SELECT t.name AS table_name, n.name AS node_name, su.used_bytes FROM sys.storage_used su JOIN sys.tables t ON t.id = su.table_id JOIN sys.nodes n ON n.id = su.node_id ORDER BY t.name, n.name; ``` *Output* ```none Text theme={null} +-------------------------+-----------+--------------+ | table_name | node_name | used_bytes | |-------------------------+-----------+--------------| | table0 | lts0 | 48000000 | | table0 | lts1 | 11000000 | | table0 | lts2 | 79000000 | | table0 | lts3 | 73000000 | | table0 | lts4 | 85000000 | | table0 | lts5 | 50000000 | | table0 | lts6 | 51000000 | | table0 | lts7 | 41000000 | | table1 | lts0 | 66000000 | | table1 | lts1 | 77000000 | | table1 | lts2 | 73000000 | | table1 | lts3 | 43000000 | | table1 | lts4 | 76000000 | | table1 | lts5 | 51000000 | | table1 | lts6 | 71000000 | | table1 | lts7 | 105000000 | ``` The query output shows how data is distributed across your nodes. Execute the `REBALANCE` task named `rebalance_task` to reorganize all data in a balanced state using the `CREATE TASK` SQL statement. ```sql SQL theme={null} CREATE TASK rebalance_task TYPE REBALANCE; ``` For details about the syntax for creating tasks, see [Distributed Tasks](/distributed-tasks). View the status of the tasks as the rebalance operation executes using the `sys.subtasks` system catalog table. ```sql SQL theme={null} SELECT id, name, start_time, task_type, execution_type, status, details FROM sys.subtasks WHERE task_type = 'rebalance' ORDER BY start_time DESC; ``` *Output* ```none Text theme={null} +--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+ | id | name | start_time | task_type | execution_type | status | details | |--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------| | ed68a070-d7f7-4c50-9b94-84b04b4503c2 | | 2024-04-17 21:44:20.268000 | rebalance | intra_cluster_rebalance | COMPLETE | | | 9e6c3374-6e9c-4a84-9bf8-ba1db3d866eb | | 2024-04-17 21:44:20.268000 | rebalance | intra_cluster_rebalance | COMPLETE | | | aebb75d8-e066-4813-b7e8-c272e5a3b8e9 | | 2024-04-17 21:44:14.768000 | rebalance | inter_cluster_rebalance | COMPLETE | | | 5cec5095-8b3a-4868-a10d-35fe6a32e998 | | 2024-04-17 21:44:14.564000 | rebalance | NULL | COMPLETE | Task finalizing due to terminal status: COMPLETE | +--------------------------------------+--------+----------------------------+-------------+-------------------------+----------+--------------------------------------------------+ ```  The output shows when the `REBALANCE` task is finished. When the `REBALANCE` task runs, segments are in the `REBUILDING` state. After the Ocient System completes this task, all segments should transition to the `INTACT` state. Execute the query from Step 1 again to see how the system reorganized the data across nodes. ```sql SQL theme={null} SELECT t.name, n.name, su.used_bytes FROM sys.storage_used su JOIN sys.tables t ON t.id = su.table_id JOIN sys.nodes n ON n.id = su.node_id ORDER BY t.name, n.name; ``` *Output* ```none Text theme={null} +-------------------------+----------+--------------+ | name | name_1 | used_bytes | |-------------------------+----------+--------------| | table0 | lts0 | 48000000 | | table0 | lts1 | 39000000 | | table0 | lts2 | 71000000 | | table0 | lts3 | 53000000 | | table0 | lts4 | 85000000 | | table0 | lts5 | 50000000 | | table0 | lts6 | 51000000 | | table0 | lts7 | 41000000 | | table1 | lts0 | 66000000 | | table1 | lts1 | 77000000 | | table1 | lts2 | 73000000 | | table1 | lts3 | 43000000 | | table1 | lts4 | 76000000 | | table1 | lts5 | 68000000 | | table1 | lts6 | 71000000 | | table1 | lts7 | 88000000 | ``` The output shows the redistributed data. The `REBALANCE` task redistributes data across segment groups, so there can be some variance in data across nodes. ## Add Loader Nodes Adding more Loader Nodes to your system can improve the throughput of data loading and resiliency against loading failure. By having extra Loader Nodes, you can also dedicate sets of nodes for specific pipelines. ### System Considerations Before starting this process, ensure you meet the requirements in the [Prerequisites](#prerequisites) section. ### Tutorial Stop any active data pipelines. Execute this SQL statement by replacing `pipeline_name` with the name of your data pipeline. ```sql SQL theme={null} STOP PIPELINE ; ``` Create the `/var/opt/ocient/bootstrap.conf` file on the new Loader Node using this YAML example. This YAML file should follow a similar format and parameters to the `bootstrap.conf` file on your other nodes. Replace the `` with the IP address or hostname of a node running the Administrator Role on your system. If you are not using a DNS, use `nodeAddress` instead of `adminHost`. Specify a user account with System Administrator privileges for your system for the username `adminUserName` and password `adminPassword`. ```yaml YAML theme={null} adminHost: adminUserName: my_admin adminPassword: example_password ``` Start the new node by using this command. ```shell Shell theme={null} sudo systemctl start rolehostd ``` This process takes about a minute to complete. This step accepts the node into the system, but does not assign a role. Add the `streamloader` role to the new node. Execute this SQL statement by replacing `` with the new node name. ```sql SQL theme={null} ALTER NODE ADD ROLE streamloader; ``` Restart the `rolehostd` process on the replacement node by running this command at the shell terminal on the replacement node. ```shell Shell theme={null} sudo systemctl restart rolehostd ``` Restart the pipeline. Execute this SQL statement by replacing `pipeline_name` with the name of your data pipeline. Optionally, specify the Loader Node by name with the `USING LOADERS` keywords to prioritize it for this pipeline. ```shell Shell theme={null} START PIPELINE USING LOADERS ; ``` If you use the legacy LAT service, you must stop loading and copy the LAT configuration files to any new Loader Nodes. For details, see [Configure the LAT Service](/replace-nodes#step-4-configure-the-lat-service). This step is unnecessary for systems that use Ocient data pipelines for loading. ## Add SQL Nodes Adding more SQL Nodes to your system can improve query optimization and processing, particularly for aggregation and join operations. Extra SQL Nodes also provide system resiliency, especially when assigned the `admin` role. ### System Considerations Before starting this process, ensure you meet the requirements in the [Prerequisites](#prerequisites) section. ### Tutorial Create the `/var/opt/ocient/bootstrap.conf` file on the new SQL Node using this YAML example. This YAML file should follow a similar format and parameters to the `bootstrap.conf` file on your other nodes. Replace the `` with the IP address or hostname of a node running the Administrator Role on your system. If you are not using a DNS, use `nodeAddress` instead of `adminHost`. Specify a user account with System Administrator privileges for your system for the username `adminUserName` and password `adminPassword`. ```yaml YAML theme={null} adminHost: adminUserName: my_admin adminPassword: example_password ``` Start the new node by using this command. ```shell Shell theme={null} sudo systemctl start rolehostd ``` This process takes about a minute to complete. This step accepts the node into the system, but does not assign a role. Add the `sql` role to the new node. Execute this SQL statement by replacing ``with the new node name. ```sql SQL theme={null} ALTER NODE ADD ROLE sql; ``` Optionally, you can also assign the `admin` role to the SQL Node. At least one SQL Node must always fulfill this role. By default, the system assigns the `admin` role to the first SQL Node in the system. For details about the `admin` role, see [Node Configuration with the Administrator Role](/node-configuration-with-the-administrator-role). Execute this SQL statement by replacing `` with the name of your new node. ```sql SQL theme={null} ALTER NODE ADD ROLE admin; ``` Restart the `rolehostd` process on the replacement node by running this command at the shell terminal on the replacement node. ```shell Shell theme={null} sudo systemctl restart rolehostd ``` Assign the new SQL Node to a connectivity pool with the [ALTER CONNECTIVITY\_POOL](/cluster-and-node-management#alter-connectivity_pool-add-participants) statement. In this example, the statement assigns the SQL Node `sql2` to the connectivity pool `cp1` with the IP address `111.1.1.1` and port number `4050` for listening. Specify the local IP address and port number `4050` to return to the client. ```sql SQL theme={null} ALTER CONNECTIVITY_POOL cp1 ADD PARTICIPANTS( NODE sql2 LISTEN_ADDRESS '111.1.1.1' LISTEN_PORT 4050 ADVERTISED_ADDRESS 'localhost' ADVERTISED_PORT 4050); ``` ## Related Links [Replace Nodes](/replace-nodes) [Replace an NVMe Node Drive](/replace-an-nvme-node-drive) [System Catalog](/system-catalog) # Formatting Functions Source: https://docs.ocient.com/formatting-functions Reference for Ocient SQL formatting functions to convert values between text and other types using format strings, locales, padding, and precision options. ## Formatting Considerations * The format string can be any combination of characters and format patterns for each function specified in the [Date and Timestamp Formatting Patterns](#date-and-time-formatting-functions) and [Date and Timestamp Formatting Modifiers](#date-and-timestamp-formatting-modifiers) tables. * Patterns can be uppercase or lowercase, but not mixed case. YYYY or yyyy matches YYYY, but not YyYy. * Any characters in the format string that are not part of a pattern match any character in that position in TO\_DATE, TO\_NUMBER, and TO\_TIMESTAMP. * You can escape string literals that contain format patterns with quotes. For example: `YYYY"a literal part YYYY"` ## Date and Time Formatting Functions ### **Date and Timestamp Formatting Patterns** | **Pattern** | **Description** | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | Y,YYY | 4 digit year with comma | | YYYY | 4 digit year | | YYY | Last 3 digits of a year. Values between 000-599 are assumed to be the year 2XXX and 600-999 are assumed to be the year 1XXX. | | YY | Last 2 digits of a year. Values between 00-69 are assumed to be the year 20XX and 70-99 are assumed to be 19XX. | | Y | Last digit of a year. Assumed to be year 200X. | | BC, bc, AD, or ad | An era indicator BC or AD in upper or lower case | | B.C., b.c., A.D., or a.d. | An era indicator B.C. or A.D. in upper or lower case | | CC | Century value 1-99. Is ignored unless YY, Y, IY, or I are set | | MM | Month of the year value 01-12 | | MONTH | Upper case English month name | | Month | English Month name with first letter capitalized | | month | lower case English month name | | RM | Upper case month of year in Roman numerals | | rm | Lower case month of year in Roman numerals | | MON | 3 letter upper case abbreviation of an English month name | | Mon | 3 letter abbreviation of an English month name with first letter capitalized | | mon | 3 letter lower case abbreviation of an English month name | | W | Week of month 1-5. For example: Days 15-21 are in the third week of the month. | | DD | Day of month 1-31 | | WW | Week of year 1-53 | | DDD | Day of year 1-366 | | D | Day of week 1-7 where Sunday is 1. This value is ignored outside of to\_char. | | IYYY | 4 digit ISO-8601 week-numbered year. See [ISO week date](https://en.wikipedia.org/wiki/ISO_week_date) for more details. | | IYY | Last 3 digits of an ISO-8601 week-numbered year. Values between 000-599 are assumed to be the year 2XXX and 600-999 are assumed to be the year 1XXX. | | IY | Last 2 digits of an ISO-8601 week-numbered year. Values between 00-69 are assumed to be the year 20XX and 70-99 are assumed to be 19XX. | | I | Last digit of an ISO-8601 week-numbered year. Assumed to be the year 200X. | | IDDD | Day of an ISO-8601 week-numbered year 001-371 | | IW | Week of an ISO-8601 week-numbered year 01-53 | | ID | Day of an ISO-8601 week 01-07 where Monday is 1 | | HH or HH12 | Hour of day 01-12 | | HH24 | Hour of day 0-23 | | AM, am, PM, or pm | Meridiem indicator AM or PM in upper or lower case | | A.M., a.m., P.M., or p.m. | Meridiem indicator A.M. or P.M. in upper or lower case | | SSSS | Seconds after midnight 0-86399 | | MI | Minute of hour 0-59 | | SS | Second of minute 0-59 | | MS | Milliseconds 000-999 | | US | Microseconds 000000-999999 | | NS | Nanoseconds 000000000-999999999 | | J | Julian Day (days after November 24, 4714 BC at midnight) | The following modifiers can also be applied to a format pattern. ### **Date and Timestamp Formatting Modifiers** | **Modifier** | **Description** | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | FM | Has no effect on TO\_DATE and TO\_TIMESTAMP. Will not be considered a literal, but will not otherwise affect the behavior. | | TH | Upper case ordinal number suffix. For example: `'DDTH'` | | th | Lower case ordinal number suffix. For example: `'DDth'` | | FX | Global prefix. By default, TO\_DATE and TO\_TIMESTAMP collapse all spaces to a single space. If FX is specified, spaces are not collapses. FX must be at the beginning of the format string to be applied. For example: `'FX YYYY-MM-DD'` | | TM | Has no effect on TO\_DATE and TO\_TIMESTAMP. Will not be considered a literal, but will not otherwise affect the behavior. | | SP | Has no effect on TO\_DATE and TO\_TIMESTAMP. Will not be considered a literal, but will not otherwise affect the behavior. | ### Date and Time notes * You can use time specifiers such as HH12 in `TO_DATE`. Their format will be validated, but their values will not affect the resulting date. * ISO 8601 formats for dates cannot be mixed with traditional formats for years, months, and days in `TO_DATE` and `TO_TIMESTAMP` * Values for MS, US, and NS are scaled up if they do not have leading zeros. `TO_TIMESTAMP('0.3', 'S.NS')` corresponds to 300000000 NS and `TO_TIMESTAMP('0.000000003', 'S.NS')` corresponds to 3 NS. * Conflicting information for the same pattern will cause an exception. ex: `TO_DATE('05 31', 'DD DD')` * Conflicting information across different patterns such as DD and DDD will generally give precedence to the pattern that appears higher in the table. ### TO\_TIMESTAMP Converts a character value with the specified format to a `TIMESTAMP` type. **Syntax** ```sql SQL theme={null} TO_TIMESTAMP(character_value, character_format) ``` | **Arguments** | **Data** **Type** | **Description** | | ------------------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `character_value` | `CHAR` | A string of a value to be converted to a `TIMESTAMP` type. The string must match the provided `character_format`. | | `character_format` | `CHAR` | The format pattern to be used to convert the `character_value`. See the [Date and Timestamp Formatting Patterns](#date-and-timestamp-formatting-patterns) section for more information about accepted pattern values. | **Examples** ```sql SQL theme={null} SELECT TO_TIMESTAMP('2022/12/01', 'YYYY/MM/DD'); ``` *Output*: `2022-12-01 00:00:00.000` ```sql SQL theme={null} SELECT TO_TIMESTAMP('2023-02-28 13:43:20.403', 'YYYY/MM/DD HH24:MI:SS'); ``` *Output*: `2023-02-28 13:43:20.000` ### TO\_DATE Converts a character value with the specified format to a `DATE` type. **Syntax** ```sql SQL theme={null} TO_DATE(character_value, character_format) ``` | **Arguments** | **Data** **Type** | **Description** | | ------------------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `character_value` | `CHAR` | A string of a value to be converted to a `DATE` type. The string must match the provided `character_format`. | | `character_format` | `CHAR` | The format pattern to be used to convert the `character_value`. See the [Date and Timestamp Formatting Patterns](#date-and-timestamp-formatting-patterns) section for more information about accepted pattern values. | **Example** ```sql SQL theme={null} SELECT TO_DATE('2022/12/01', 'YYYY/MM/DD'); ``` *Output*: `2022-12-01` ## Number Formatting Functions ### **Number Formatting Patterns** | **Pattern** | **Description** | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | 9 | A digit. | | 0 | A digit. In TO\_CHAR, includes leading zeros. | | . or D | A decimal point. | | , or G | A group/thousand separator. | | PR | Negative value in braces. For example: `-1 = <1>` | | S or SG | A + or - sign for positive or negative, respectively. | | MI | A - sign if the number is negative. | | PL | A + sign if the number is positive. | | V | Shifts the decimal place by the number of digits after V in the format. Cannot be mixed with `.` or `D`. For example: `TO_NUMBER('32', '9V99')` => 0.32 | | RN | Roman numeral. Not supported by TO\_NUMBER. | | EEEE | Exponent for scientific notation. Not supported by TO\_NUMBER. | | L | Currency symbol. | ### Number Formatting Notes * You cannot mix PR with other sign indicators. * Any character that does not match the corresponding group in the format string is ignored. `TO_NUMBER('1&2', '999')` = 12. * If PR is present, it must be after every 0 or 9 in the format string. * Similar to TO\_DATE and TO\_TIMESTAMP, the FM is not considered a literal, but will not otherwise affect the behavior of TO\_NUMBER. ### TO\_NUMBER Converts a character value with the specified format to a `DECIMAL` type. **Syntax** ```sql SQL theme={null} TO_NUMBER(character_value, character_format) ``` | **Arguments** | **Data** **Type** | **Description** | | ------------------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `character_value` | `CHAR` | A string of a value to be converted to a `DECIMAL` type. The string must match the provided `character_format`. | | `character_format` | `CHAR` | The format pattern to be used to convert the `character_value`. See the [Number Formatting Patterns](#number-formatting-patterns) section for more information about accepted pattern values. | **Example** ```sql SQL theme={null} SELECT TO_NUMBER('5230.87', '9999V99'); ``` *Output*: `5230.87` ## Related Links [Date and Time Functions](/date-and-time-functions) [Time Zone Functions](/time-zone-functions) [Character and Binary Functions](/character-and-binary-functions) # Frequently Asked Questions for Data Pipelines Source: https://docs.ocient.com/frequently-asked-questions-for-data-pipelines Frequently asked questions about Ocient data pipelines, covering setup, deduplication, error handling, monitoring, performance, and source-format support. The questions featured in this topic cover information that you need most often when you start creating and running data pipelines. ## Running Pipelines ### Who can run a data pipeline? Any database user with `EXECUTE` pipeline privileges can run a data pipeline on their database. This privilege allows database administrators to distinguish the users who can create pipelines from those who can start them. To learn more about access control for pipelines, see [DCL Pipeline Privileges](/data-control-language-dcl-statement-reference). ### Do pipelines run automatically after creation? No, pipelines have a user-controlled lifecycle that starts with creation. After you create a data pipeline, you determine when to start the pipeline. A pipeline either completes successfully or moves to a failed status if the System reaches an error limit. For a basic batch file load, you would first use the `CREATE PIPELINE` SQL statement and then the `START PIPELINE` statement. After all data is successfully loaded, the pipeline status changes to completed. ### How do you run multiple pipelines at the same time? Multiple data pipelines can run concurrently in Ocient. To run more than one simultaneously, any user with privileges to start a pipeline can use the `START PIPELINE` SQL statement. Execution of this statement begins the execution of the selected pipeline. The Ocient System uses Loader Nodes in parallel to process all of the pipelines that are running. If you need to control the relative priority of one pipeline over another when running concurrently, use advanced pipeline settings to limit the number of cores allocated to a specified pipeline or isolate the pipeline to a particular set of Loader Nodes. ### What happens if I start a pipeline that has been stopped? When you execute a `START PIPELINE` SQL statement on a pipeline that has been stopped due to either hitting an `ERROR LIMIT` or due to user intervention, the load resumes from the last point in the pipeline. Hence, you can effectively pause a pipeline and resume from the last checkpoint in the pipeline. This feature works for continuous pipelines such as pipelines as well as batch pipelines. Common uses of stopping and resuming a pipeline are: * Stopping to update the transformations in a continuous pipeline. * Stopping to update the table schema and refresh a pipeline. * Stopping to allow a higher priority workload to proceed. * Stopping for maintenance. It is important to exercise caution when resuming a pipeline to ensure that the files in the underlying data source have not changed. Modifying data in the source for an in-process pipeline can result in unexpected results, such as duplicate data. ## File Loading ### When is the file list determined for a batch file load? When you load a batch of files in a data pipeline, the Ocient System determines the batch when the pipeline starts using the bucket, the prefix, and the filter parameters you specify in your pipeline. This batch of files is frozen until the pipeline completes. Even if the batch pipeline stops and restarts, the Ocient System keeps the list of files constant. ### Can I update the file list after starting a pipeline? You cannot manually modify the file list for a batch pipeline. When a pipeline restarts, it finishes the files that were determined when the batch was first established. To add files that arrive on the file system after a batch pipeline was first started, you can create a new batch pipeline that uses filters to load all files created after a specified timestamp. ### What is the behavior if I start a pipeline that has been stopped? When you start a file-based pipeline that has been stopped, the pipeline adds any new files to the pipeline as files with the `PENDING` status. The pipeline then loads these files as execution proceeds. The Ocient System does not reload any files that the pipeline has already loaded. It is important to exercise caution when resuming a pipeline to ensure that the files in the underlying data source have not changed. When you modify data in the source for an in-process pipeline, the Ocient System can have unexpected results, such as duplicate data. ### What options exist to maximize performance on S3-type file loads? The most common performance enhancement for S3 file loads is the `PREFIX` parameter in your pipeline. This setting restricts the files in the bucket that your pipeline considers, speeding up listing operations. You can also use other advanced options such as `MAX_CONCURRENCY`, `REQUEST_DEPTH`, and `REQUEST_RETRIES` described in [S3 Source Options](/data-pipelines#s3-source-options). ## Kafka Loading ### How does a continuous Kafka pipeline work? Continuous pipelines for Kafka stream records from Kafka partitions and ensure exactly-once semantics are delivered. Ocient assigns consumers to different Loader Nodes and commits offsets back on the assigned partitions as the source data loads into Ocient. This operation critically only occurs after data is durable in Ocient, which means that the system replicates data in such a way that the loss of a configurable number of nodes would not lead to data loss. All supported data formats are compatible with Kafka pipelines. When you run a continuous Kafka pipeline, the pipeline is never complete so it remains in a running state. You can observe progress using the standard system catalog tables and the Kafka-specific table `sys.pipeline_partitions` that captures individual partition assignments and lag. When you stop and restart a Kafka pipeline, it resumes from the last commit checkpoint and continues processing records. The Ocient System deduplicates any record that has already been sent to Ocient but not committed to Kafka in the Loader Nodes. ### How do I update the transforms or the table schema in a continuous Kafka pipeline? To update transforms in a Kafka pipeline, first, stop the pipeline. After the pipeline stops, you can execute a `CREATE OR REPLACE PIPELINE` SQL statement. This action allows you to update the transformations in the pipeline and maintains all of the checkpoint information about progress on the pipeline. When you execute the `START PIPELINE` SQL statement, the pipeline resumes from the last Kafka committed offsets. Then, the Loader Nodes deduplicate any record that has already been sent to the Loader Nodes but is not yet committed to Kafka. ### Where does a Kafka pipeline resume loading after a stop and then start of the pipeline? When you execute the `START PIPELINE` SQL statement, the pipeline resumes from the last Kafka committed offsets. The Loader Nodes deduplicate any record that has already been sent to the Loader Nodes but is not yet committed to Kafka. If you have never started a Kafka pipeline, the pipeline starts from the configured `auto.offset.reset` value. You can set this value in the `CONFIG` option of a Kafka pipeline. The Ocient System uses this value when a pipeline first starts to establish the consumer group for the pipeline. For more details about Kafka configuration, see [Kafka Source Options](/data-pipelines#kafka-source-options). ## Data Duplication and Deduplication ### What records are deduplicated during loading? Ocient Loader Nodes deduplicate data that is loaded on the same pipeline. You can think of the deduplication scope of a pipeline as being tied to the life of the pipeline database object. If you drop a pipeline and create a new one with the same name, this data is *not* deduplicated. For a specified pipeline, the Ocient System deduplicates data automatically when you stop and resume a pipeline. The system ensures that no duplicate data is loaded for any reprocessed data due to managing watermarks with data sources such as Kafka or S3. It is important to note that deduplication in Ocient is based on a shared contract between the data source and the Ocient System. Deduplication is not based on a primary key or record identifier that is part of the data itself. The [What mistakes can accidentally lead to duplicating data?](#what-mistakes-can-accidentally-lead-to-duplicating-data) question explains the implications more clearly. ### How can I load the same data set twice into the same table? Sometimes, it is useful to load data multiple times for testing, but deduplication can get in the way. To load the same data set twice in the same table, there are two simple approaches: 1. Drop the pipeline from the first load, create the same pipeline, and run it to completion. 2. Create a new pipeline with the same SQL statement as the original, but with a new name, and run it to completion. In both cases, the Ocient System does not deduplicate the data, and you can load a second copy of your data. ### What mistakes can accidentally lead to duplicating data? The Ocient System can have duplicate data when the deduplication contract between the source and the pipeline is not maintained. Ocient uses a proprietary approach to deduplication that helps it deliver high throughput when loading while still ensuring exactly-once semantics. If the data in the source changes during the execution of a pipeline, the changes can lead to a mismatch in the way the Ocient System determines the deduplication identifier for a record. Things to avoid include: * Modifying a file in the source file system after it has been included in a pipeline, but the pipeline has not been completed. * Deleting a file in the source that has been included in a pipeline, but the pipeline has not been completed. You can add new files safely to a pipeline before it completes, but you should avoid modifying files already registered by the pipeline and appear in the `sys.pipeline_files` system catalog table. ## Troubleshooting and Errors ### How do I detect when an entire file fails in a pipeline? If an entire file has failed, you can see this in the `sys.pipeline_files` system catalog table. Any file with the `FAILED` status indicates that the file failed to process. This failure can occur due to a file-level error, such as corruption. In addition, if a file has the `SKIPPED` status, this status indicates that the file was skipped when processing the pipeline. You can control this behavior using the `FILE_ERROR` setting in the `START PIPELINE` SQL statement. If a specified file experiences record-level errors, but the Ocient System processes the file, the file has the `LOADED_WITH_ERRORS` status. ### How do I find the individual records that failed to load? You can find the individual errors in the `sys.pipeline_errors` system catalog table. You can determine if a pipeline encounters errors by viewing the `information_schema.pipeline_status` view that includes counts of records with errors. ## Transformations ### How can I convert a string representation of an array of data into an array data type? The Ocient System automatically converts a string representation of an array into an array using the settings defined on your pipeline. The system uses the `CLOSE_ARRAY`, `OPEN_ARRAY`, and `ARRAY_ELEMENT_DELIMITER` settings to convert a string to a target array column type. These types default to a -style array (e.g., `{1,2,3}`). For details, see the supported options in [Extract Options](/data-pipelines#general-extract-options). ### How do I apply a transformation function to an array of data? Data pipelines do not yet support the mapping of arbitrary transformation functions over an array. However, the Ocient System supports casting and type conversion. You can apply an array cast to an array and the system applies the type of each element as a conversion to the elements of the array. For details, see [Scalar Transformation Functions and Casting](/transform-data-in-data-pipelines#scalar-transformation-functions-and-casting). ### My data is in a non UTF-8 encoding. How do I get the load to decode my character set properly? You can change the character set of your pipeline using the `CHARSET_NAME` option in a pipeline. For details, see the supported options in [Extract Options](/data-pipelines). ### Can a data pipeline extract nested data in JSON? Ocient supports a JSON selector syntax similar to JSON Path. The Ocient System can extract individual keys as well as nested data. A simple example is `$order.user.first_name`, which extracts the first name `first_name` from the `user` object on the `order` object in JSON. For details, see [Supported JSON Selectors](/data-formats-for-data-pipelines#supported-json-selectors). ### Can a data pipeline extract attributes of objects in an array in JSON? Ocient supports a JSON selector syntax similar to JSON Path. The Ocient System can extract individual keys, nested objects, arrays, and arrays of objects. The system can also project a selector into the arrays. A simple example is `$data.line_items[].price`, which extracts a one-dimensional array of prices `price` from the `line_items` array inside the `data` object. For details, see [Supported JSON Selectors](/data-formats-for-data-pipelines). ### How do I apply a user-defined transformation to the data? Data Pipelines support the creation of user-defined functions that you can use to apply more advanced transformations. For details, see the [CREATE PIPELINE FUNCTION](/data-pipelines#create-pipeline-function) SQL statement. ### What automatic conversions does Ocient perform from my source data to the target columns? The Ocient System supports many automatic conversions to make it easy to load source data into your target tables. `VARCHAR` data automatically casts into most target column types. The system casts many data types automatically to a `VARCHAR` target column. Some other common automatic conversions, such as `BIGINT` to `TIMESTAMP`, are supported, especially where the conversion is lossless. For details, see [Data Types for Data Pipelines](/data-types-for-data-pipelines). ### How do I cast data to an array column? To cast to an array column, use a casting function. This function converts a string of data in the configured array format to the array type you specify, casting the elements to the inner type. For example, if `$my_column` represents the data `'{123,456,789}'` and your pipeline is using the default `CLOSE_ARRAY`, `OPEN_ARRAY`, and `ARRAY_ELEMENT_DELIMITER` settings, this expression snippet casts the elements to an array of `INT` values. Ocient also automatically casts `$my_column` in this example to an `INT[]` when that is the type of the target column. ```sql SQL theme={null} CREATE PIPELINE ... SELECT INT[]($my_column) as column_array_of_ints; ``` ### How do I cast data to a tuple column? Similar to casting to an array column, you can cast to a tuple column using tuple casting syntax. For example, if `$my_column` represents the data `'(123,test)'`, and your pipeline is using the default `CLOSE_TUPLE`, `OPEN_TUPLE`, and `TUPLE_ELEMENT_DELIMITER` settings, this expression snippet casts the elements as a tuple of an integer and a string `TUPLE<>`. ```sql SQL theme={null} CREATE PIPELINE ... SELECT TUPLE<>($my_column) as column_tuple; ``` ## Related Links [Load Data](/load-data) [Data Pipelines Reference](/data-pipelines) # Functions Overview Source: https://docs.ocient.com/functions-overview Explore Ocient function library for data analysis, manipulation, and transformations, empowering complex querying and calculations. includes many functions for statistics, mathematics, string manipulation, and working with complex types like Arrays and geospatial data. The following types of functions are supported: ## SQL Functions ### Math and Arithmetic Functions Perform mathematical operations on data in Ocient, including basic operators on data types and advanced statistical functions. * [Math Functions and Operators](/math-functions-and-operators) * [Aggregate Functions](/aggregate-functions) ### Character and Binary Functions Manipulate character strings and binary data types and use character operators to query character data. * [Character and Binary Functions](/character-and-binary-functions) ### Conversion and Formatting Functions Convert data types with scalar conversions and format numbers and times in character outputs. * [Scalar Data Conversion Functions](/scalar-data-conversion-functions) * [Formatting Functions](/formatting-functions) ### Date and Time Functions Perform date operations in Ocient, including manipulation and comparison of date and time types. Date and Time functions allow users to add, subtract, and truncate date time values and intervals. * [Date and Time Functions](/date-and-time-functions) * [Time Zone Functions](/time-zone-functions) ### Aggregates Aggregates include basic aggregates and sorted aggregates that combine many rows into a single result, as well as window aggregates that combine information from multiple rows within the window into each row in the result set. * [Aggregate Functions](/aggregate-functions) * [Window Aggregate Functions](/window-aggregate-functions) * [HyperLogLog Functions](/hyperloglog-functions) ### Array, Tuple, and Matrix Functions Special functions and operators to work with container types. These include operators to check set logic, slice, compare, and manipulate arrays, tuples, and matrices. * [Array Functions and Operators](/array-functions-and-operators) * [Tuple Functions and Operators](/tuple-functions-and-operators) * [Matrix Functions and Operators](/matrix-functions-and-operators) ### Geospatial Functions Functions and Operators to manipulate and compare points, linestrings, and polygons. * [Geospatial Functions](/geospatial-functions) ### Network Functions Functions for using network data types, including `IP` and `IPV4`. * [Network Type Functions](/network-type-functions) ### Miscellaneous Functions A set of functions such as `CASE`, `CURRENT_USER`, and `NULL_IF` that provide additional capabilities in Ocient. * [Other Functions and Expressions](/other-functions-and-expressions) ## Related Links [SQL Reference](/sql-reference) [Data Query Language (DQL) Statement Reference](/data-query-language-dql-statement-reference) # Generate Tables Using sys.dummy Source: https://docs.ocient.com/generate-tables-using-sys-dummy Use the sys.dummy table in Ocient to generate sample rows for testing queries, prototyping SQL, demonstrating functions, and learning syntax patterns. `sys.dummy` is a virtual table that the generates with a specific number of rows. This table is a handy tool to quickly create tables for testing without scripting extensive DDL statements. When you execute `sys.dummy`, the database creates a table with a single column, which is automatically populated with incremental integers. `sys.dummy` follows similar rules to a table reference. The SQL statement must specify a non-negative integer that represents the number of rows for the SQL statement to generate in the table. In this syntax block, the `{N}` parameter represents the number of rows for `sys.dummy` to generate. `SELECT *` queries that lack a `FROM` clause automatically reference the `sys.dummy1` table. For example, `SELECT *;` is the same as `SELECT * FROM sys.dummy1;`. **Syntax** ```sql SQL theme={null} sys.dummy{N} ``` | **Parameter** | **Description** | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `{N}` | The number of rows for `sys.dummy` to generate.
The minimum numeric value for `{N}` is `0`, and the maximum is `9223372036854775807`. This parameter must not have any leading or trailing characters. If `{N}` does not adhere to these limits, the database returns an error.
Do not specify curly brackets with this parameter in queries. | **Example** Create a table with 10 rows. ```sql SQL theme={null} SELECT * FROM sys.dummy10; ``` \*Output: \* ```sql SQL theme={null} c1 --- 1 2 3 4 5 6 7 8 9 10 ``` ## Related Links [Data Query Language (DQL) Statement Reference](/data-query-language-dql-statement-reference) [SQL Reference](/sql-reference) # Geospatial Functions Source: https://docs.ocient.com/geospatial-functions Reference for Ocient OcientGeo geospatial functions, including constructors, measurement, spatial relationships, and spatiotemporal operators for GIS data. The has that supports three different geospatial geographies: `POLYGON`, `LINESTRING`, and `POINT`. * A `POLYGON` can be constructed with a closed LINESTRING or an outer shell and array of inner rings. * A `LINESTRING` represents a series of points connected by line segments. It can be constructed with either `LINESTRING` or `POINT` data types. * A `POINT` represents a point in space defined by an (x, y) or (longitude, latitude) coordinate pair. Each `LINESTRING` or `POLYGON` value can be up to a maximum of 512 MB in size. This means a `LINESTRING` or `POLYGON` can contain approximately 32 million point values. These supported geospatial data types require Well-Known Text (WKT) formatting. For formatting examples, see [Data Types](/data-types) or [WKT Representation of Geometry](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Geometric_objects). The OcientGeo functionality reference also contains examples with supported formatting and syntax. Each geospatial geography has a number of functions and operators that can be used in SQL queries to perform analyses. Some functions apply to specific geography types. For operators specific to geospatial data types, refer to [Geospatial Operators](#geospatial-operators). OcientGeo uses the authalic radius of the Earth, 6371007.1810824 meters, as the reference for many measurement calculations. The authalic radius reference might cause slight differences in measurements compared to other geospatial systems that use a different radius definition. ## Geospatial Operators OcientGeo supports special operators that perform operations on two different geographies. These can be used in queries to compare geographies or calculate values based on the geographies. | **Operator** | **Syntax** | **Purpose** | | --------------------- | ----------------- | -------------------------------------------------------------------------------------------- | | intersects | `geoA && geoB` | Returns `TRUE` if the bounding box of geography A intersects with the bounding box of B. | | contained | `geoA @ geoB` | Returns `TRUE` if the bounding box of geography A is contained within the bounding box of B. | | equal | `geoA ~= geoB` | Returns `TRUE` if the bounding box of geography A is the same as bounding box of B. | | contains | `geoA ~ geoB` | Returns `TRUE` if the bounding box of geography A contains the bounding box of B. | | distance | `geoA <-> geoB` | Returns the distance in meters between the two geography parameters. | | bounding box distance | `geoA <#> geoB` | Returns the distance in meters between the bounding box of the two geography parameters. | | centroid distance | `geoA <<->> geoB` | Returns the distance in meters between the centroids of the two geography parameters. | ## Geospatial Functions Overview OcientGeo supports many geospatial functions that operate on POINT, LINESTRING, and POLYGON values. Each function, its expected inputs, and return type are described in the following section. ## Geospatial Filtering Queries that involve geospatial filtering can make use of functions in the [Spatial Relationships](/spatial-relationships) section. To optimize these queries, OcientGeo provides a `SPATIAL` index that you can apply to columns with geospatial data types. For details about the `SPATIAL` index, see [Secondary Indexes](/secondary-indexes#spatial-index-type). ## [Attribute Functions](/attribute-functions) Attribute functions return descriptive information on the specified data set. * ST\_COORDDIM * ST\_DIMENSION * ST\_GEOMETRYTYPE * ST\_ISEMPTY * ST\_MEMSIZE * ST\_NDIMS * ST\_NDIMENSION * ST\_NPOINTS * ST\_NUMPOINTS * ST\_SRID * ST\_X * ST\_XMAX * ST\_XMIN * ST\_Y * ST\_YMAX * ST\_YMIN ## [Conversion Functions](/conversion-functions) Conversion functions transform a specified operand to a different data type. * ST\_ASBINARY * ST\_ASWKB * ST\_ASGEOJSON * ST\_ASEWKT * ST\_ASLATLONTEXT * ST\_ASTEXT * ST\_ASWKT * ST\_GEOHASH ## [Linestring Constructors](/linestring-constructors) LINESTRING constructors use geospatial data to create a LINESTRING object. * ST\_LINEFROMTEXT * ST\_LINEFROMGEOJSON * ST\_LINEFROMWKB * ST\_LINEFROMEWKT * ST\_LINESTRING * ST\_MAKELINE ## [Linestring Functions](/linestring-functions) LINESTRING functions can perform alterations or access descriptive information on LINESTRING objects. * ST\_ADDPOINT * ST\_ENDPOINT * ST\_LINEINTERPOLATEPOINT * ST\_LINELOCATEPOINT * ST\_LINESUBSTRING * ST\_POINTN * ST\_REMOVEPOINT * ST\_SETPOINT * ST\_STARTPOINT ## [Point Constructors](/point-constructors) POINT constructors use geospatial data to create a POINT object. * ST\_CENTROID * ST\_GEOGPOINT * ST\_MAKEPOINT * ST\_POINT * ST\_POINTFROMEWKT * ST\_POINTFROMGEOHASH * ST\_POINTFROMGEOJSON * ST\_POINTFROMTEXT * ST\_POINTFROMWKB ## [Polygon Constructors](/polygon-constructors) POLYGON constructors use geospatial data to create a POLYGON object. * ST\_FORCECCW * ST\_MAKEPOLYGON * ST\_POLYGON * ST\_POLYGONFROMGEOJSON * ST\_POLYGONFROMWKB * ST\_POLYGONFROMEWKT * ST\_POLYGONFROMTEXT * ST\_WHOLEEARTH ## [Spatial Measurement](/spatial-measurement) Spatial measurement functions can perform basic calculations on geospatial data, such as measuring the distance between two POINT objects or the area of a POLYGON object. * ST\_AREA * ST\_ANGLE * ST\_AZIMUTH * ST\_DISTANCE * ST\_DISTANCESPHERE * ST\_DISTANCESPHEROID * ST\_EUCLIDEANDISTANCE3D * ST\_HAUSDORFFDISTANCE * ST\_LENGTH * ST\_LENGTH2D * ST\_MINIMUMDISTANCETOSURFACE * ST\_MAXDISTANCE * ST\_PERIMETER * ST\_PERIMETER2D ## [Spatial Operators](/spatial-operators) Spatial operators perform geometry calculations on geospatial data to return a different type of geospatial data. * ST\_BOUNDINGDIAGONAL * ST\_BUFFER * ST\_CLOSESTPOINT * ST\_CONVEXHULL * ST\_DIFFERENCEARRAY * ST\_ENVELOPE * ST\_EXPAND * ST\_EXTERIORRING * ST\_FLIPCOORDINATES * ST\_FORCE2D * ST\_INTERIORRINGN * ST\_INTERSECTALL * ST\_INTERSECTIONARRAY * ST\_LONGESTLINE * ST\_MAKEENVELOPE * ST\_MINIMUMBOUNDINGCIRCLE * ST\_MULTIDIFFERENCEARRAY * ST\_MULTIINTERSECTIONARRAY * ST\_MULTISYMDIFFERENCEARRAY * ST\_MULTIUNIONARRAY * ST\_NRINGS * ST\_NUMINTERIORRING * ST\_NUMINTERIORRINGS * ST\_POINTONSURFACE * ST\_PROJECT * ST\_REDUCEPRECISION * ST\_REMOVEREPEATEDPOINTS * ST\_REVERSE * ST\_SEGMENTIZE * ST\_SHORTESTLINE * ST\_SIMPLIFY * ST\_SIMPLIFYARRAY * ST\_SNAPTOGRID * ST\_SYMDIFFERENCEARRAY * ST\_UNIONARRAY ## [Spatial Relationships](/spatial-relationships) Spatial relationship functions use arguments to test for different types of spatial relationships.  * ST\_CLUSTERDBSCAN * ST\_CONTAINS * ST\_CONTAINSPROPERLY * ST\_COVERS * ST\_COVEREDBY * ST\_CROSSES * ST\_DISJOINT * ST\_DWITHIN * ST\_EQUALS * ST\_INTERSECTS * ST\_ISCCW * ST\_ISPOLYGONCCW * ST\_ISPOLYGONCW * ST\_ISCLOSED * ST\_ISRING * ST\_ISSIMPLE * ST\_ISVALID * ST\_OVERLAPS * ST\_POINTINSIDECIRCLE * ST\_RELATE * ST\_TOUCHES * ST\_WITHIN ## [Spatiotemporal Measurement](/spatiotemporal-measurement) Spatiotemporal measurement functions can perform basic calculations on geospatial data paired with TIMESTAMP data. * ST\_DISTANCE * ST\_MAXDISTANCE * ST\_TOTALSECONDSININTERSECTION ## [Spatiotemporal Operators](/spatiotemporal-operators) Spatiotemporal operators perform calculations on geospatial data using an array of TIMESTAMP arguments. * ST\_LONGESTLINE * ST\_LINEGETALLTIMESATPOINT * ST\_LINEGETPOINTATTIME * ST\_LINEGETTIMEATPOINT * ST\_INTERSECTION * ST\_SHORTESTLINE ## Related Videos [At the Whiteboard with Ocient: Geospatial Analytics](https://youtu.be/tIlTpbCP0Rg) # Global Dictionary Compression Source: https://docs.ocient.com/global-dictionary-compression Use global dictionary compression in Ocient to reduce storage costs on repeated string values and accelerate query performance with dictionary-based filters. Global Dictionary Compression (GDC) is a feature of the database that compresses variable length column data using a dictionary encoder. Instead of storing the variable length data directly on disk, GDC seamlessly substitutes an integer that corresponds to each unique string. ## Advantages of GDC 1. **Reduced disk usage** — A string like `"GDC is really cool!"` takes up 19 raw bytes. With GDC, it takes at most 4 bytes. This benefit is multiplied by the number of rows in the table and, when applied to arrays, can lead to a dramatic reduction in required storage. 2. **Fixed-width** — Variable length column data can be more difficult to perform selective input or output (I/O). It is much faster for to read the 100th element in a list of integers by offsetting 100 times the fixed width of the integer. With variable-length types, a full traversal is required, counting how many elements are passed until the 100th is found. This can improve performance in some queries. 3. **Faster Joins** — When joining on a GDC column, equality comparisons can be performed on the GDC integers instead of the variable-length data. This can improve performance in some queries. 4. **Allows Variable-Length Cluster Keys** — As of version 19.0 of the Ocient System, GDC is the only method by which a variable-length column can be used as a cluster key index. ## Disadvantages of GDC 1. **Increased complexity** — GDC does add some configuration complexity. It requires a user to know the rough cardinality of their data to size the GDC number of bytes and adds system configuration options. 2. **Load complexity** — GDC compression adds some overhead to loading data into the Ocient System. 3. **Not suitable for high-cardinality data** — If the variable length column has higher cardinality than this, it should not be stored using GDC. ## GDC Syntax To create a GDC column, add the `COMPRESSION GDC(int)` parameter after the column type specifier when creating a column, where `int` is one of 1, 2, or 4. The integer specifies the number of bytes to be used for the integer keys stored on the disk. This also corresponds to the maximum number of unique keys to be stored in a particular column. | **GDC(int)** | **Maximum Number of Unique Values** | | ------------ | ----------------------------------- | | GDC(1) | 256 | | GDC(2) | 65,536 | | GDC(4) | 4,294,967,296 | There is a *soft* limit of 1,000,000 keys even on 4-byte integer GDC columns. Contact Ocient Support to evaluate your criteria for changing the limit and to understand the impact of raising limits. ### Apply GDC to a Column To enable GDC on a column, the keyword is applied in the `CREATE TABLE` or `ALTER TABLE ADD COLUMN` statement. An example of GDC on a `VARCHAR` column: `{column_name} VARCHAR(255) COMPRESSION GDC(2)` When you apply GDC to one column, the Ocient System creates a view instead of a table from the full column definition. ### Reuse an Existing GDC Map A user can also specify that a column should share GDC space with another column–possibly even a column of a different table. This would be useful if the same data is used in multiple columns and it is often joined in queries. Use `COMPRESSION GDC EXISTING schema.table.column_name` as shown here: `{column_name} VARCHAR(255) GDC EXISTING {schema.table.column_name}` ### GDC on Array Columns For arrays of variable-length data, GDC operates on the individual elements of the array. Specify compression after the overall array type: `{column_name} VARCHAR(255)[] COMPRESSION GDC(2)` ### GDC on Tuple Columns Elements of tuple columns can be compressed with GDC. Specify compression on the specific type to be compressed: `{column_name} tuple` ## GDC in the System Catalog You can use the system catalog to inspect the number of keys used by different columns and the maximum count on each. **Example Query:** ```sql SQL theme={null} SELECT t.id as table_id, t.name as table_name, c.id as column_id, c.name as column_name, g.compressed_size, g.current_count, g.max_count FROM sys.tables t INNER JOIN sys.global_map_table_info g ON t.id = g.table_id INNER JOIN sys.columns c ON c.id = g.column_id; ``` ## Truncation When a table with GDC is partially truncated, *GDC key mappings for removed rows are not removed*. This can result in stale mappings. The only way to remove unwanted mappings is to drop the column and recreate it. ## GDC Column Representation in the System Catalog To the end user, a table with GDC columns looks like any other table. However, when creating a table with GDC columns, the table configuration is different in system catalog tables. Instead of a table named `schema.tablename`, GDC tables leverage a built-in view. This view and several pieces of metadata will be created in the system catalog: 1. A view named `schema.tablename` is added in `sys.views`. This view is the representation of the user of the GDC table, and it automatically converts the GDC keys to the loaded variable-length data. This is the table that the user interacts with for querying data, making alterations, and granting or revoking access. 2. For each GDC column in the table, a table named `syslookup.schema_tablename_columnname` is added in `sys.tables`. These tables store the mappings from strings to integers for each column. These tables should be interacted with only rarely. 3. A table called `sysgdc.schema_tablename` is added in `sys.tables`. This is the table that is stored to disk, including all the non-GDC columns and the integers for each GDC column. This table should be interacted with only rarely. ## GDC Columns and Loading into the Database GDC stores key mappings in a Raft consensus log that is maintained by all nodes operating with the Admin role. These mappings are created as the data is loaded, and used in queries to fetch the correct variable length data for the stored integers. Mappings can also be created by certain queries if the key does not already exist. Because the Admin roles are responsible for maintaining the GDC key mappings, there must be a consensus of Admin roles available on the system in order to load new data that includes GDC columns. ## Related Links [System Catalog](/system-catalog) # Glossary Source: https://docs.ocient.com/glossary Glossary of acronyms and terms used in the Ocient System, covering architecture, SQL, data pipelines, geospatial, and ML concepts. Use this table to reference frequently used acronyms and terms with the System or the . | **Name** | **Definition** | | -------- | ------------------------------------- | | AA | Auto Acquire | | AdTech | Advertising technology | | ANTLR | ANother Tool for Language Recognition | | API | Application Programming Interface | | ASIO | Async I/O library | | AST | Abstract Syntax Tree | | CFL | Continuous File Loading | | CmdComp | Command Compiler | | CTAS | CREATE TABLE AS SELECT SQL statement | | CTE | Common Table Expression | | DCL | Data Control Language | | DDL | Data Definition Language | | DML | Data Manipulation Language | | DR | Disaster Recovery | | ELT | Extract, Load, and Transform | | ETL | Extract, Transform, and Load | | FinTech | Financial technology | | GDC | Global Dictionary Compression | | GDS | Global Data Storage | | GIS | Geospatial Information System | | HA | High Availability | | IAS | INSERT INTO SELECT SQL statement | | IDA | Information Dispersal Algorithm | | LCK | Local Cluster Key | | LTS | Foundation Node | | NUMA | Non-Uniform Memory Architecture | | NVMe | Non-Volatile Memory Express | | OLAP | Online Analytical Processing | | OSC | Ocient Stream Client | | OSN | Ownership Sequence Number | | TKT | Time | | TTQ | Time to Queryability | | UDT | User-defined Transform | | UIO | User mode I/O | | UUID | Universally Unique Identifier | | VM | Virtual Machine | | WLM | Workload Management | # Google Cloud Platform Ocient Installation Source: https://docs.ocient.com/google-cloud-platform-ocient-installation Set up an Ocient system on Google Cloud Platform, utilizing cloud infrastructure for scalable and efficient data warehousing. This guide explains how to install an System in (GCP). For details about GCP concepts, see these pages: * [What is Compute Engine?](https://cloud.google.com/compute/docs/concepts) * [What is IAM?](https://cloud.google.com/iam/docs/overview) * [VPC Network Overview](https://cloud.google.com/vpc/docs/overview) Ocient supports deployment in GCP for pilot or testing purposes, but this setup does not guarantee data durability. Stopping Compute Engine virtual machine (VM) instances can result in permanent data loss. The steps for deploying an Ocient System in GCP are: 1. Prepare GCP resources. 2. Set up an initial instance. 3. Create machine images from the initial instance. 4. Launch other instances. 5. Follow the standard Ocient installation procedure. ## **Example Configuration** The table below shows the recommended VM types for each node type. | **Node Type** | **VM Type** | | -------------------- | ------------- | | Foundation Nodes (3) | n2-highmem-96 | | Loader Nodes (1) | n2-highmem-96 | | SQL Nodes (1) | n2-highmem-96 | Metadata and non-Ocient nodes use the n2-standard-X machine type, where X represents the number of Virtual Central Processing Units (vCPUs) for your workload. This diagram shows an example of an Ocient cluster in GCP. The system deploys the Compute Engine VMs (SQL, Loader, and Foundation) within a single subnet of a Virtual Private Cloud (VPC). GCP uses VPC firewall rules and network tags to control traffic to each node type. Virtual private cloud for loading data from a Cloud Storage bucket using a JDBC client ## Prepare GCP Resources Create and configure these GCP resources: * The VPC network and subnets for the Ocient System. For details, see [Create and manage VPC networks](https://cloud.google.com/vpc/docs/create-modify-vpc-networks). * VPC firewall rules with network tags to control access to the endpoints for each node type. For details about firewall configuration, see [Create firewall rules](https://cloud.google.com/vpc/docs/using-firewalls) and [Add and remove network tags](https://cloud.google.com/vpc/docs/add-remove-network-tags). For details about the network security configuration, see the [Ocient Security Guide](/ocient-security-guide). * Identity and Access Management (IAM) roles. For details, see [Grant an IAM role](https://cloud.google.com/iam/docs/granting-changing-revoking-access). If you are loading data from Cloud Storage, the Loader Nodes require IAM access to a Cloud Storage bucket. Assign the `roles/storage.objectViewer` role (or `roles/storage.objectAdmin` if you also need write access) to the service account used by the Loader Node VMs. ## Node Setup (SQL Role) for the Initial Instance Use this configuration for your machine image. Configuration steps differ depending on whether your setup uses a single-volume or multi-volume machine image. For details about creating instances, see [Create and start a Compute Engine instance](https://cloud.google.com/compute/docs/instances/create-start-instance). ### **Operating System (OS)** To set up the machine image, you can use any Ocient-supported OS (see [Ocient System Requirements](/ocient-system-requirements)). **Single-Volume Machine Image** If you use a single-volume machine image, specify this configuration: * Increase the boot disk to 128GB or more. **Multi-Volume Machine Image** If you use a multi-volume machine image (e.g., CIS hardened (RHEL) 9), use this configuration: * Increase the boot disk to 30GB or more. * Increase the additional Persistent Disk volume to 100GB or more. For details, see [Add a Persistent Disk to your VM](https://cloud.google.com/compute/docs/disks/add-persistent-disk). This Persistent Disk volume supports key system directories in the image (`/home`, `/var`, `/var/log`, `/var/log/audit`, `/var/tmp`). **VM Type** Use n2-highmem-96 for Ocient nodes. Use n2-standard-X for Metadata and non-Ocient nodes. **Secure Boot** Disable Shielded VM secure boot when creating the instance. Use the `--no-shielded-secure-boot` flag with the `gcloud` command. ```shell Shell theme={null} gcloud compute instances create INSTANCE_NAME \ --no-shielded-secure-boot ``` If you are using the Console, uncheck **Turn on Secure Boot** under **Security > Shielded VM**. For details, see [Shielded VM](https://docs.cloud.google.com/compute/shielded-vm/docs?hl=en). **Firewall Rules and Network Tags** Create VPC firewall rules with network tags that enforce these rules: * Allow SSH to the nodes. * Allow communication internally between nodes. * Allow access to SQL Node endpoints. For details, see the [Ocient Security Guide](/ocient-security-guide). Assign the appropriate network tags to each VM so that the firewall rules apply to the correct node types. For details, see [Create firewall rules](https://cloud.google.com/vpc/docs/using-firewalls) and [Add and remove network tags](https://cloud.google.com/vpc/docs/add-remove-network-tags). Connect to your instance using Secure Shell (SSH). For details, see [Connect to Linux VMs using Google tools](https://cloud.google.com/compute/docs/connect/standard-ssh). This step applies only to multi-volume machine images. If you are using a single-volume instance, skip this step. If you use a multi-volume manager, extend the `/home` and `/var` LVM volumes and their filesystems to fill up the expanded Persistent Disk volumes. These actions expand the LVM volume and the contained file system to accommodate the package, logging, and metadata of the Ocient System. These code examples show how to extend LVM volumes for a RHEL 9 image. Other machine image types might require different sizing. Contact Ocient support for the best sizing for your system for multi-volume instances. **Examples** Resize the physical volumes of two drives to use their full capacity after expanding them (see [Prepare GCP Resources](#prepare-gcp-resources)). ```shell Shell theme={null} sudo pvresize /dev/nvme0n1 sudo pvresize /dev/nvme1n1 ``` Extend local volumes: * Add 66 percent of the available free space in `vg-01` to the `var_vol` logical volume. * Add all (100 percent) of the remaining free space in `vg-01` to the `home_vol` logical volume. ```shell Shell theme={null} sudo lvextend -l +66%FREE /dev/vg-01/var_vol sudo lvextend -l +100%FREE /dev/vg-01/home_vol ``` Extend the file system to use all available space on its underlying logical volume. ```shell Shell theme={null} sudo xfs_growfs /home sudo xfs_growfs /var ``` Update all your software packages to their latest versions and then reboot your instance. For RHEL-compatible systems, use this command. ```shell Shell theme={null} sudo dnf update ``` For -compatible systems, use this command. ```shell Shell theme={null} sudo apt update ``` Reboot after the update. ```shell Shell theme={null} sudo reboot ``` GCP Compute Engine VMs require additional kernel modules for full hardware support. Install the extra kernel modules package that matches your running kernel version. ```shell Shell theme={null} sudo apt install linux-modules-extra-$(uname -r) ``` Reboot the node after the installation completes. ```shell Shell theme={null} sudo reboot ``` This step is required for GCP Compute Engine VMs. Without the extra kernel modules, certain hardware drivers and features might not function correctly. Copy over the `ocient` RPM or DEB package and install it. For RHEL-compatible systems, use this command. ```shell Shell theme={null} sudo dnf install ./ocient-RELEASE-XX.X.X-xxxxxxxxxxxxxx-x86_64.rpm ``` For Debian-compatible systems, use this command. ```shell Shell theme={null} sudo apt install ./ocient-RELEASE-XX.X.X-xxxxxxxx.xxxxxx-xxxxxxxxxxxx-amd64.deb ``` Use the `ockernelparams` utility to set up kernel parameters automatically, including the huge pages configuration. Repeat this step on other nodes. ```shell Shell theme={null} sudo /opt/ocient/scripts/ockernelparams --node-role sql ``` Reboot the system for the parameters to take effect. ```shell Shell theme={null} sudo reboot ``` Check that the local storage drive is attached to the `uio` or `vfio` driver after reboot (the Ocient package installs a service that runs on startup to perform this action). ```shell Shell theme={null} sudo /opt/ocient/scripts/nvme-driver-util.sh ``` On GCP, drives bound to the `uio_pci_generic` driver display without a device name. This behavior is expected for GCP NVMe drives. ```text theme={null} NVMe device status BDF Numa Node Driver name Device name 0000:00:04.0 -1 uio_pci_generic - ``` For examples of attaching drivers to the NVMe drives, see [NVMe Drive Firmware Upgrade Process](/nvme-drive-firmware-upgrade-process#page-title). If you are not using an OS-level firewall, skip this step. If your base machine image includes a system firewall, you must configure rules that explicitly allow required network communication for your Ocient deployment. For details, see the [Ocient Security Guide](/ocient-security-guide). Required OS firewall rules: * Allow all necessary ports and protocols between Ocient nodes by: * Opening all TCP/UDP ports within the private network range (e.g., `10.0.0.0/16`), or allowing known Ocient ports. * Allow external access where needed by: * Enabling SSH access (port `22`) from your administrator IP range. * Allowing client access to SQL endpoints (for example, port `13101` or as specified in your setup). * Opening any additional ports required for monitoring or management tools. For a list of required ports, see [Network Exposure and Firewall](/ocient-security-guide#network-exposure-and-firewall). In addition to OS-level firewall rules, ensure that your GCP VPC firewall rules and network tags are properly configured (see [Prepare GCP Resources](#prepare-gcp-resources)). Both layers must allow the required traffic for the Ocient System to function correctly. ## Create the Machine Image After your initial node is fully configured, you must replicate the setup process for the remaining nodes in your cluster. This action ensures consistency and allows for proper internal communication between nodes. For details about creating a machine image, see [Create a machine image](https://cloud.google.com/compute/docs/machine-images/create-machine-images). ## Set Up Remaining Nodes to Launch the Remaining Instances Go through this process for each of your remaining nodes. Launch the remaining instances with these parameters. For details, see [Create an instance from a machine image](https://cloud.google.com/compute/docs/machine-images/create-instance-from-machine-image). * Machine image — Use the machine image created in the [Node Setup (SQL Role) for Initial Instance](#node-setup-sql-role-for-initial-instance) step. * VM type — Use `n2-highmem-96` for Ocient nodes. This VM type: * Provides sufficient memory and compute for high-performance Ocient workloads. * Has high throughput and network bandwidth for internal cluster communication. * Secure boot — Disable secure boot using the `--no-shielded-secure-boot` flag. * VPC firewall rules — Ensure these rules are in place using network tags associated with the nodes: * Allow SSH to the nodes. * Allow internal communication between all Ocient nodes. * Allow access to endpoints described in the [Ocient Security Guide](/ocient-security-guide). Connect to your instance using SSH. For details, see [Connect to Linux VMs using Google tools](https://cloud.google.com/compute/docs/connect/standard-ssh). Use the `ockernelparams` utility to automatically set up kernel parameters, including the `hugepages` parameters. This example specifies a Foundation Node `foundation`. Use a different node type as necessary. ```shell Shell theme={null} sudo /opt/ocient/scripts/ockernelparams --node-role foundation ``` Reboot the system for the parameters to take effect. ```shell Shell theme={null} sudo reboot ``` ## Bootstrap the Ocient System Complete the bootstrapping process for your Ocient System. For details, see [Node Bootstrapping Reference](/node-bootstrapping-reference). ## Related Links [Load Data](/load-data) [Query Ocient](/query-ocient) # Guide to Rebuilding Segments Source: https://docs.ocient.com/guide-to-rebuilding-segments Step-by-step guide to rebuilding segments in an Ocient System using distributed tasks to restore redundancy after disk replacement or node failure events. Data in an System is erasure coded to provide resilience to disk and Foundation Node failures. The resilience of a system depends on the parity width of the storage space, which represents the number of associated disks (or nodes) that can fail without interruption of service or data loss. For details about storage spaces and parity width, see [Configure Storage Spaces](/configure-storage-spaces). ## Data Segment Statuses You can check the status of your system segment groups by querying the `sys.segment_groups` system catalog table. For details, see [System Catalog](/system-catalog). This table describes the states for data segments. | **Status** | **Description** | **Recovery Process** | | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | INTACT | The normal and operable status. | No recovery needed. | | DAMAGED | The segment failed a checksum, meaning the segment data is corrupted and unusable. | If you have sufficient parity width, you can recover a damaged segment by invoking a rebuild segment task. | | MISSING | The segment is on a node or disk that is currently offline. If the node or disk rejoins the cluster, it can transition to the INTACT status. | When a disk or Foundation Node is permanently removed, you can perform a rebuild task to recover the data. This requires sufficient parity width. | | REBUILDING | The segment is in recovery after damage or missing segment data. | Recovery is already in progress. | ## Recovery Considerations If a segment has the DAMAGED or MISSING status, queries can proceed by reconstructing the data on demand using the remaining erasure-coded data in the segment group. Having a non-INTACT status means that input or output (I/O) performance is significantly reduced. To restore full performance, you need to run a segment rebuild task. Segment rebuilding is not automatic. The system administrator must manually invoke it. A segment rebuild fails, and data is lost completely if the number of segments with the DAMAGED status in a segment group exceeds the parity width of the storage space. To avoid data loss: * Provision the parity width of the storage space at or above the number of expected concurrent node failures. * Rebuild damaged or permanently missing segments as soon as possible. ## Checking for Abnormal Segments You can find any segments that need a rebuild by querying for segment groups with an abnormal status. **Examples** **Finding Faulty Segment Groups** This example query finds any segment groups with the DAMAGED or MISSING status. ```sql SQL theme={null} SELECT * FROM sys.segment_groups WHERE status IN ('DAMAGED', 'MISSING'); ``` *Output* ```none Text theme={null} | "id" | "cluster_id" | "segment_type" | "status" | "primary_owner" | "loader_id" | "table_id" | "scope_id" | "block_size" | "begin_time" | "end_time" | "coding_algorithm" | "coding_block_size" | "coding_threshold" | "coding_width" | "replication" | "parity_cycle" | "created_time" | "rolehostd_version" | "commit_hash" | "timestamp" | "build_user" | "depth" | "removal_type" | |----------------------|----------------------------------------|----------------|----------|-----------------|----------------------------------------|----------------------------------------|----------------------------------------|--------------|--------------|------------|--------------------|---------------------|--------------------|----------------|---------------|----------------|-------------------------|---------------------|--------------------------------------------|-------------------|--------------|---------|----------------| | "720576045199095878" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT | DAMAGED | | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | bd18d33b-aae9-4be2-a2f4-76ecdc34c2ba | "8bc119d2-875a-47e4-b016-87fde83e77d6" | 4096 | 0 | 1 | XOR_PARITY | 4096 | 2 | 3 | 1 | 1 | 2025-02-26 20:23:34.274 | "25.0.0" | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" | | 0 | NOT_REMOVED | | "716072445571725367" | "074c32ec-4f92-4718-ada1-5eb55eafdcb3" | TKT_SEGMENT | DAMAGED | | "9a8f7ea7-613c-499f-ab2d-937ab0ce992e" | "9fe79009-ee75-4429-b29b-3a614a166751" | a8c805d0-2144-46a7-8374-52cb88d64244 | 4096 | 0 | 1 | XOR_PARITY | 4096 | 2 | 3 | 1 | 1 | 2025-02-26 20:23:19.447 | "25.0.0" | "91ab76ae57491ade0122bc7b594d5c3c6e0bf40c" | "20250109.221519" | | 0 | NOT_REMOVED | ``` **Finding Clusters and Nodes with Faulty Segment Groups** Inspect the count of damaged groups by cluster and node. ```sql SQL theme={null} SELECT c.name AS cluster_name, n.name AS node_name, g.status AS segment_group_status, seg.status AS segment_status, seg.kind, COUNT(*) AS segment_count FROM sys.segment_groups g LEFT JOIN sys.clusters c ON c.id = g.cluster_id LEFT JOIN sys.stored_segments seg ON seg.segment_group_id = g.id LEFT JOIN sys.nodes n ON n.id = seg.node_id WHERE g.status <> 'INTACT' AND (seg.status <> 'INTACT' OR seg.status IS NULL) GROUP BY 1,2,3,4,5 ORDER BY 1,2,3,4,5; ``` *Output* ```none Text theme={null} | "cluster_name" | "node_name" | "segment_group_status" | "segment_status" | "kind" | "segment_count" | |--------------------|-------------|------------------------|------------------|---------|-----------------| | foundation_cluster | foundation0 | DAMAGED | | VIRTUAL | 2 | ``` ## Starting a Segment Rebuild Task A user with System Administrator privileges can start a segment rebuild using the `CREATE TASK TYPE REBUILD` SQL statement. You cannot cancel a segment rebuild task after it is started. Most commonly, a rebuild task repairs all damaged or missing segments across the system. **Example** Create a rebuild task. ```sql SQL theme={null} CREATE TASK TYPE REBUILD; ``` The system can continue to perform queries while rebuilding segments, but the process can impact I/O performance. ### Advanced Rebuild Commands Rebuild tasks can also execute on specific Foundation Nodes or clusters. For information on fine-tuning rebuild tasks, see [CREATE TASK](/distributed-tasks#create-task). ## Checking Rebuild Task Status Monitor the status of current and past segment rebuild tasks from the `sys.subtasks` system catalog table. For details, see [System Catalog](/system-catalog). ```sql SQL theme={null} SELECT * FROM sys.subtasks WHERE task_type = 'rebuild'; ``` This table describes the statuses for a rebuild task. | **Status** | **Status Detail** | **Description** | **Next Steps** | | ---------- | ---------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | complete | no\_work | The segment groups were already available and healthy. No rebuilding was needed. | None | | complete | complete | Rebuild completed successfully. | None | | running | rebuild\_in\_progress | Rebuild is in process. | You can monitor the progress of the rebuild task by checking the JSON dictionary in the `details` column of the `sys.subtasks` system catalog table. | | failed | rebuild\_not\_possible | The number of missing or damaged segments exceeds the parity width, and the data cannot be recovered currently. | If missing segments are available on an offline drive, you can attempt another rebuild task when that drive is made available to the system.
Otherwise, you cannot recover the data. | | failed | rebuild\_no\_space | There is not enough space available to rebuild the segment. | You can complete the rebuild by truncating other data to free up space. | | failed | failed\_on\_node | The cluster lost its connection to the node before the rebuild was completed. | This is a transient error. You can retry the rebuild task. | | failed | error | An unexpected internal error occurred. | Review error message details using the `rolehostd` logs. For details, see [Log Monitoring](/log-monitoring). | ## Related Links [CREATE TASK](/distributed-tasks) [Erasure Coding](/compute-adjacent-input-and-output-on-large-working-sets#erasure-coding) # HyperLogLog Functions Source: https://docs.ocient.com/hyperloglog-functions Reference for HyperLogLog functions in Ocient SQL for fast, approximate distinct counting at scale, including merge and cardinality estimation operations. The HyperLogLog (HLL) sketch family of functions provides an approximate count of the number of unique elements in one or more columns. HLL functionality is similar to running a query using a `COUNT(DISTINCT col)` clause and uses the same mechanisms as the [aggregate function](/aggregate-functions) `APPROX_COUNT_DISTINCT`. The tradeoff for HLL sketches is accuracy. For further explanation of this tradeoff in accuracy, see the [DataSketches documentation](https://datasketches.apache.org/docs/Background/TheChallenge.html). Each sketch is an approximate and compact representation of the original data, and it introduces a small margin of error. The HLL implementation is based on the algorithm outlined in the research paper [HyperLogLog in Practice](https://research.google/pubs/pub40671/). This implementation uses `log2k = 11` by default, which provides a percent error of approximately 5% at the 95% confidence interval. ## HLL Sketch Functions in the SQL Statement Sketches are useful on aggregated tables that require estimates of distinct counts. This example creates a table for the new sketch columns `ip_address_sketch` and `user_id_sketch` as part of a CREATE TABLE AS SELECT statement. The new table inserts values from the `master_agg_table` table. ```sql SQL theme={null} CREATE TABLE my_sketch_table ( col_a INT, col_b VARCHAR(255), ip_address_sketch HLL_SKETCH(11), user_id_sketch HLL_SKETCH(11) ) AS ( SELECT col_a, col_b, HLL_SKETCH_CREATE(ip_address) as ip_address_sketch, HLL_SKETCH_CREATE(user_id) as user_id_sketch FROM master_agg_table GROUP by 1,2); ``` After the database creates the new table, you can query for the approximate count of distinct values. This example merges the sketch columns by using an `HLL_SKETCH_UNION` function. After the database merges the columns, the database converts sketch values to a distinct count estimate by using the `HLL_SKETCH_GET_ESTIMATE` function. ```sql SQL theme={null} SELECT HLL_SKETCH_GET_ESTIMATE(HLL_SKETCH_UNION(ip_address_sketch)) AS ip_address_sketch, HLL_SKETCH_GET_ESTIMATE(HLL_SKETCH_UNION(user_id_sketch)) AS user_id_sketch FROM my_sketch_table; ``` ## HLL Accuracy with log2k Parameter In addition to the default `log2k = 11` implementation, you can specify a `log2k` value in the range of \[10, 16] when creating an HLL Sketch. This parameter controls the number of buckets used in the HLL algorithm, which controls the accuracy of the sketch upper bounded by `1.04 / sqrt(k)`. The tradeoff with larger `log2k` values is that the sketch size grows exponentially. See this table for specifics on how the precision values affect size and accuracy. | **Precision** | **Uncompressed Size** | **95% CI** | | ------------- | --------------------- | ---------- | | 10 | 1032 B | ±6.50% | | 11 (default) | 2056 B | ±4.60% | | 12 | 4104 B | ±3.25% | | 13 | 8200 B | ±2.30% | | 14 | 16392 B | ±1.63% | | 15 | 32776 B | ±1.15% | | 16 | 65544 B | ±0.81% | By default, `log2k` values larger than 11 are compressed on disk using ZSTD compression. When creating or referencing a HLL Sketch column in a `CREATE TABLE`, `CTAS` or `IAS` statement, you can use the type alias `HLL_SKETCH(log2k)` instead of the internal `HASH((2^log2k) + 8)`. This type alias is supported in `EXPORT` statements as well. **Example** In this example, a CREATE TABLE statement creates two HLL Sketch columns with specified `log2k` precision values. * `sketch_a` has a precision value of `10`, meaning it has low precision and low storage requirements. * `sketch_b` has a precision value of `15`, meaning it has high precision and high storage requirements. Note that the precision values specified in both HLL\_SKETCH\_CREATE statements match the precision values in the CREATE TABLE columns. ```sql SQL theme={null} CREATE TABLE my_sketch_table ( sketch_a HLL_SKETCH(10), sketch_b HLL_SKETCH(15) ) AS ( SELECT HLL_SKETCH_CREATE(col_a, 10) as sketch_a, HLL_SKETCH_CREATE(col_b, 15) as sketch_b FROM master_agg_table GROUP by 1,2); ``` ## Supported HLL Sketch Functions ### HLL\_SKETCH\_CREATE Creates an HLL sketch from the data on a specified aggregated column. Returns a `HASH((2^log2k) + 8)` data representation of the sketch that you can store in a separate column. The HLL algorithm depends on the hash value of its input. When merging multiple sketches, it is important that each sketch derives from the same type in order to get accurate results. `1::BIGINT` might hash differently than `1::INT`, and thus, the HLL algorithm might not view these two hashes as referring to the same value `1`. For accurate results, users should cast numeric columns or expressions to a consistent type before they are used in a `HLL_SKETCH_CREATE` function, especially if they will be merged later on. **Syntax** ```sql SQL theme={null} HLL_SKETCH_CREATE(agg_col, [ log2k ] ) ``` | **Arguments** | **Data** **Type** | **Description** | | ------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `agg_col` | All data types are supported. | An aggregated column for use when you create a sketch. | | `log2k` | Integral literal | Optional.
This parameter controls the number of buckets in the HLL Sketch, which affects the accuracy and storage of the sketch. This value must be an integer literal in the range of \[10, 16]. It cannot be a reference to a column.
By default, `log2k` values larger than 11 are compressed on disk using ZSTD compression.
For details about this parameter, see [HLL Accuracy with log2k Parameter](#hll-accuracy-with-log2k-parameter). | **Example** This example uses `HLL_SKETCH_CREATE` as a window aggregate to generate 100 sketches that contain one value in each sketch. ```sql SQL theme={null} SELECT MOD(c1, 9) AS m, HLL_SKETCH_CREATE(c1, 12) OVER (PARTITION BY c1) AS sketch FROM sys.dummy100; ``` ### HLL\_SKETCH\_UNION (aggregate function) Merges multiple sketches in a single column into a unified sketch. All sketches must have the same precision. This function is an aggregate function and operates on a column. **Syntax** ```sql SQL theme={null} HLL_SKETCH_UNION(agg_sketch_col) ``` | **Arguments** | **Data** **Type** | **Description** | | ---------------- | ------------------- | ---------------------------------------------------------------------- | | `agg_sketch_col` | `HLL_SKETCH` | A column that contains multiple sketches to be merged into one sketch. | **Example** This example performs two merges. First, this code uses `HLL_SKETCH_CREATE` as a window aggregate to generate 100 sketches that contain one value in each sketch. The code merges the 100 sketches into nine sketches, and then the final SELECT statement merges the nine sketches into a single sketch. Also, the example uses the `HLL_SKETCH_GET_ESTIMATE` function to retrieve the estimated distinct count from the merged sketches. ```sql SQL theme={null} WITH hll_sketch AS ( SELECT MOD(c1, 9) AS m, HLL_SKETCH_CREATE(c1) OVER (PARTITION BY c1) AS sketch FROM sys.dummy100 ), hll_merged AS ( SELECT HLL_SKETCH_UNION(hll_sketch.sketch) AS sketch FROM hll_sketch GROUP BY m ) SELECT HLL_SKETCH_GET_ESTIMATE(HLL_SKETCH_UNION(hll_merged.sketch)) FROM hll_merged; ``` *Output*: `97` ### HLL\_SKETCH\_UNION (scalar function) Merges two sketches to a new combined sketch. This function is a scalar function and operates row-wise. The scalar function merges two sketch columns with heterogeneous precisions into a sketch with the lower of the two precisions. **Syntax** ```sql SQL theme={null} HLL_SKETCH_UNION(sketch_col1, sketch_col2) ``` | **Arguments** | **Data** **Type** | **Description** | | ------------- | ------------------- | ------------------------ | | `sketch_col1` | `HLL_SKETCH` | A column of HLL sketches | | `sketch_col2` | `HLL_SKETCH` | A column of HLL sketches | **Example** This example creates two sketches from the dummy table. The first uses the default precision of 11, and the second uses a precision of 14. The second sketch is offset by 1000, so there are 2000 unique values being sketched overall. The query merges the two sketches using the two argument scalar union, which combines them into a new sketch. This unified sketch has a precision of 11, the lower of the two input precisions. The query then decodes the sketch, which has the expected result of 2000 ± 92. ```sql SQL theme={null} SELECT HLL_SKETCH_GET_ESTIMATE( HLL_SKETCH_UNION( HLL_SKETCH_CREATE(c1), HLL_SKETCH_CREATE((c1 + 1000)::INT, 14) ) ) as sketch_col FROM sys.dummy1000; ``` *Output*: `2000 ± 92` ### HLL\_SKETCH\_GET\_ESTIMATE The `HLL_SKETCH_GET_ESTIMATE` scalar function converts a sketch into a distinct count estimate of a sketch value. Returns the distinct count estimate as a `BIGINT`. To return a distinct count across a column of sketch values, you should first merge the sketches by using the `HLL_SKETCH_UNION` function. **Syntax** ```sql SQL theme={null} HLL_SKETCH_GET_ESTIMATE(sketch) ``` | **Arguments** | **Data** **Type** | **Description** | | ------------- | ------------------- | ----------------------------------------------------------- | | `sketch` | `HLL_SKETCH` | A sketch value to convert to an approximate distinct count. | **Example** ```sql SQL theme={null} WITH hll_sketch AS ( SELECT MOD(c1, 9) as m, HLL_SKETCH_CREATE(c1) OVER (PARTITION BY c1) AS sketch FROM sys.dummy100 ), hll_merged AS ( SELECT HLL_SKETCH_UNION(hll_sketch.sketch) AS sketch FROM hll_sketch GROUP BY m ) SELECT HLL_SKETCH_GET_ESTIMATE(HLL_SKETCH_UNION(hll_merged.sketch)) FROM hll_merged; ``` *Output*: `97` ### HLL\_SKETCH\_GET\_ESTIMATE\_BOUND The `HLL_SKETCH_GET_ESTIMATE_BOUND` scalar function takes a `HLL_SKETCH` column or an integral `log2k` literal value and returns the resulting bounding 95-percent confidence interval error proportion as a `DOUBLE`. If you provide an integral `log2k` value, it must be an integral literal and cannot be a column. **Syntax** ```sql SQL theme={null} HLL_SKETCH_GET_ESTIMATE_BOUND(sketch_or_integral) ``` | **Arguments** | **Data** **Type** | **Description** | | -------------------- | -------------------------------------------- | ----------------------------------------------- | | `sketch_or_integral` | `HLL_SKETCH`
or integral literal | A sketch value, column, or an integral literal. | **Example** ```sql SQL theme={null} SELECT HLL_SKETCH_GET_ESTIMATE_BOUND(HLL_SKETCH_CREATE(c1, 14)) FROM sys.dummy100; ``` *Output*: `0.0163` ### HLL\_SKETCH\_TO\_STRING The `HLL_SKETCH_TO_STRING` scalar function takes a `HLL_SKETCH` column or value and returns a string summary of the sketch. **Syntax** ```sql SQL theme={null} HLL_SKETCH_TO_STRING(sketch) ``` | **Arguments** | **Data** **Type** | **Description** | | ------------- | ------------------- | -------------------------------------------------- | | `sketch` | `HLL_SKETCH` | A sketch value or column to summarize as a string. | **Example** ```sql SQL theme={null} SELECT HLL_SKETCH_TO_STRING(HLL_SKETCH_CREATE(c1)) AS summary FROM sys.dummy10000; ``` *Output*: ```none Text theme={null} ### HLL SKETCH SUMMARY: Log2k : 11 Lower bound : 9540.000000 Estimate : 10000 Upper bound : 10460.000000 Number of values seen : 10000 Minimum index : 114 Value at minimum index : 0 Maximum index : 1892 Value at maximum index : 14 ``` This summary contains this information: * The `log2k` value. For more information, see [HLL Accuracy with log2k](#hll-accuracy-with-log2k-parameter). * The lower, estimate, and upper bound of the sketch. * The number of values seen in the sketch. * The minimum and maximum bucket indexes and their values in the sketch. The output fields in this summary are for internal purposes, except for the `log2k` value. ## Bibliography Heule, Stefan, Marc Nunkesser, and Alex Hall. “HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm.” In *Proceedings of the EDBT 2013 Conference*. Genoa, Italy, 2013. ## Related Links [Aggregate Functions](/aggregate-functions) [Generate Tables Using sys.dummy](/generate-tables-using-sys-dummy) # Identifiers Source: https://docs.ocient.com/identifiers Learn about Ocient identifier rules for naming tables, columns, and other SQL elements, ensuring consistency and clarity in database design. In the System, identifiers are lexical tokens that name specific entities in the database. For example, a table name is an identifier. In most cases, you do not have to quote an identifier. However, the identifier must meet these two conditions: * Start with a letter followed by zero or more letters, digits, or underscores. * Must not be a reserved word. For the full list of reserved words, see [Reserved Words](/reserved-words). If the identifier does not meet either of these conditions, then enclose the identifier in double quotes `""` to create a delimited identifier. SQL is case-insensitive, and the database normalizes identifiers to lowercase. For example, you must add double quotes to the `1start_with_digit`, `hy-phen`, and `mixedCase` identifiers so that the database can parse them correctly: `"1start_with_digit"`, `"hy-phen"`, and `"mixedCase"`. The system uses identifiers to identify these objects: * Database name * Schema name * Function name * Table name * Table alias name * Column name * Column alias name * Index name * View name * Task name * Node names * Machine learning model name * Connectivity pool name * Connectivity pool participant name * Storage space name * Cluster name * Service class name * Single sign-on (SSO) protocol * Tag name * JSON key * Username * Group name * Common table expression * Subexpression name * Data pipeline name * Data pipeline type ## Related Links [SQL Syntax Conventions](/sql-syntax-conventions) [SQL Reference](/sql-reference) [Machine Learning Model Functions](/machine-learning-model-functions) [Connectivity Pools](/cluster-and-node-management#connectivity-pool) [Users, Groups, and Service Classes](/users-groups-and-service-classes) # Get Started with the OcientAIQ Unified Data Platform Source: https://docs.ocient.com/index Welcome to Ocient documentation. Find setup guides, SQL references, machine learning, geospatial features, integrations, and operations for the platform. The is a petabyte-scale data platform that brings AI, analytics, and high-performance query execution directly to enterprise data. Rather than requiring you to move, copy, or sample large data sets across fragmented systems, the platform unifies data ingest, query optimization, security, and governance in a single platform. Agents, applications, and analysts connect to the platform to work with governed enterprise data in place. The platform supports both structured and semi-structured data, giving AI and analytical workloads access to richer context without additional data pipelines. The platform continuously ingests high-volume data so that it is available for queries and AI-driven workflows as it arrives. Built-in access controls, auditability, and compliance features protect data across every interaction — whether the consumer is a human analyst, a BI tool, or an AI agent. You can deploy the platform in the environment that fits your operational, security, and data-sovereignty requirements. The following sections introduce the core concepts that define how data is stored, organized, and executed within the OcientAIQ Unified Data Platform. Core concepts and terms needed for a quick start Installation, Upgrade, System Configuration, Maintenance, and Monitoring of an Ocient System Connection to Ocient with JDBC and other drivers Overview of key database administration capabilities like workload management, result set caching, and managing users, groups, and roles Loading and transformation capabilities, managing pipelines, transformation functions, and error handling Overview of Ocient data types and the components of a query Overview of Ocient System design and architecture These sections outline the capabilities of Ocient for data ingestion, transformation, and advanced analytics. Ocient SQL supports integrated libraries for geospatial analysis and machine learning tools. SQL command, function, operator, and keyword reference including DDL, DCL, and query reference Different SQL client connectors Tutorials to deploy various ML models integrated into Ocient Functions for analyzing geospatial data sets For AI tools, start with [/llms.txt](https://docs.ocient.com/llms.txt) for optimal content extraction. For information about integrations and connections to third-party applications and tools, see [Ocient Integrations](/ocient-integrations). For a detailed explanation of the in Ocient, design principles, storage capabilities, and query engine, see [Ocient Architecture](/ocient-architecture). For information on ingesting data using the legacy Loading and Transformation (LAT) functionality, see [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference). # Indexes Source: https://docs.ocient.com/indexes Manage indexes in Ocient with SQL DDL, including CREATE INDEX and DROP INDEX, plus guidance on cluster keys and secondary index strategies. This group of DDL SQL statements allows database administrators to manage indexes. Database administrators can create and drop indexes. You can view information about indexes using the `sys.indexes` system catalog table. For information on other database components, see the pages on [Databases](/databases), [Schemas](/schemas), [Tables](/tables), and [Views](/views). ## CREATE INDEX `CREATE INDEX` creates a new secondary index. Indexes help optimize database queries when created on columns that are frequently referenced. For more information on how indexes operate, see [Secondary Indexes](/secondary-indexes). Creating an index does not trigger re-indexing of existing segments. Only segments generated after the `CREATE INDEX` is issued contain the new index. Indexes can be created on columns containing various different data types as long as the requirements are met. Please note that depending on the data type, the system can assign different index types by default if you decline to specify which index type to use. The name must be distinct from the name of any existing index on the table. You can apply indexes regardless of whether they have GDC compression. **Syntax** ```sql SQL theme={null} CREATE INDEX [ IF NOT EXISTS ] index_name ON table (column_name) [ USING ] ::= INVERTED | HASH | NGRAM [ (n_value) ] | SPATIAL | ZONE_MAP ``` | **Parameter** | **Type** | **Description** | | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `index_name` | string | An identifier for the index to create.
The name must be distinct from the name of any existing index on the table. | | `table` | string | The name of the table for the index. | | `column_name` | string | The name of the column for the index.
Identical indexes on the same column are not allowed. A column can only have multiple indexes if they are of different types or parameters. | | `n_value` | integer | Optional.
When used with an NGRAM index, this numeric value specifies the character length of the substrings to be indexed.
If unspecified, this value defaults to `3`. | ### Index Types (``) Ocient supports four index types alongside the clustering index: `INVERTED`, `HASH`, `NGRAM`, and `SPATIAL`. An index notionally stores a mapping of a column value to the rows that contain that value, and the index type differentiates the format in which the column values are stored and accessed. Unless an index type is explicitly specified with a `USING` clause, the data type of a column determines a default index type that the system creates. For information on index type defaults, see [Index Type Requirements and Defaults](/secondary-indexes#index-type-requirements-and-defaults). For container data types (e.g., arrays and tuples), the index stores the internal elements of the container, and is used on predicates that target the internal values. However, a mapping of NULL column values is generally stored for both scalar and container data types, so the index can always be used for `column IS NULL` predicates. | **Index Type** | **Primary Data Types** | **Primary Usage Description** | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `INVERTED` | Fixed-length numeric columns. | Stores whole column value internally, meaning its storage size is approximately the same as the width of the data type.
Supports lookups using strict equality or range comparisons.
For details, see the [INVERTED index](/secondary-indexes#inverted-index-type) section. | | `HASH` | Variable-length character columns | Stores a hash of the indexed column value rather than the full value.
Primarily used for exact comparisons.
For details, see the [HASH index](/secondary-indexes#hash-index-type) section. | | `NGRAM` | Variable-length character columns | Stores substrings equal in size to its `n_value`.
Storage requirements can greatly vary depending on column data size, width and cardinality.
Supports exact string comparison and filters including `LIKE`, `NOT LIKE`, `SIMILAR TO` and `NOT SIMILAR TO`.
For details, see the [NGRAM index](/secondary-indexes#ngram-index-type) section. | | `SPATIAL` | Geospatial columns (`POINT`, `LINESTRING`, `POLYGON`) | Groups geographic objects for bounding-box filtering.
For details, see the [SPATIAL index](/secondary-indexes#spatial-index-type) section. | | `ZONE_MAP` | Fixed-length columns
(INT, BIGINT, SMALLINT, FLOAT, DOUBLE, BINARY, DECIMAL, HASH, CHAR, IP / IPV4, TIME, TIMESTAMP, DATE, UUID) | Supports range and exact matches on a whole segment level.
The system stores the minimum and maximum values for the column per-segment. | For further description and examples of the index types, see [Secondary Indexes](/secondary-indexes). **Examples** This example creates an index named `new_idx` on the `address` column of the table. Because `address` is a `VARCHAR` column, this index defaults to the HASH index type. ```sql SQL theme={null} CREATE INDEX new_idx ON employees (address); ``` This example creates an index of type NGRAM on the `address` column. As the NGRAM has no specified `n_value`, it defaults to indexing substrings of three characters long. ```sql SQL theme={null} CREATE INDEX ngram_address_idx ON employees (address) USING NGRAM; ``` This example creates an index on a component of the `tuple_col` column. As this column is of data type `INT`, the index defaults to using the `INVERTED` type. ```sql SQL theme={null} CREATE INDEX tuple_index ON employees (tuple_col[1]); ``` This example creates an index on a component of the `point_col` column. As this column is of data type `POINT`, the index defaults to using the `SPATIAL` type. ```sql SQL theme={null} CREATE INDEX spatial_index ON employees (point_col) ``` ## DROP INDEX `DROP INDEX` drops a secondary index on a table. After an index is dropped, new segments that are generated do not contain the new index. However, no existing segments will be altered. This means that until a segment is rebuilt, you can still use the removed index internally, and the system does not reclaim the storage space the removed index occupied. **Syntax** ```sql SQL theme={null} DROP INDEX [ IF EXISTS ] index_name ON table_name ``` | **Parameter** | **Type** | **Description** | | ------------- | -------- | --------------------------------------------- | | `index_name` | string | An identifier for the index to drop. | | `table_name` | string | The name of the table with the index to drop. | **Example** This example drops the index named `new_idx` on the `employees` table. ```sql SQL theme={null} DROP INDEX new_idx ON employees; ``` ## Related Links [Core Elements of an Ocient System](/core-elements-of-an-ocient-system) [Data Query Language (DQL) Statement Reference](/data-query-language-dql-statement-reference) [Database Password Security Settings](/database-password-security-settings) [System Catalog](/system-catalog) # Information Schema Source: https://docs.ocient.com/information-schema Explore the collection of system views that provide metadata about the Ocient System, such as table, data pipeline, and index information. The system catalog exposes a set of system views in the `information_schema` schema that contain metadata about the System. Most of these views follow the SQL standard with additional views specific to the system. ## Alphabetical List of Views * `information_schema.columns` * `information_schema.data_types` * `information_schema.databases` * `information_schema.geometry_columns` * `information_schema.groups` * `information_schema.index_recommendations` * `information_schema.indexes` * `information_schema.information_schema_catalog_name` * `information_schema.nodes` * `information_schema.pipeline_status` * `information_schema.pipeline_status_historical` * `information_schema.pipeline_table_metrics` * `information_schema.pipelines` * `information_schema.pipelines_historical` * `information_schema.reserved_words` * `information_schema.schemata` * `information_schema.table_privileges` * `information_schema.table_storage` * `information_schema.tables` * `information_schema.transactional_pipeline_status` * `information_schema.users` * `information_schema.views` ## System View Descriptions #### information\_schema.columns The `columns` view shows information for all columns in the system. | Column Name | Column Type | Column Description | | ----------------- | ----------- | -------------------------------------------------------- | | table\_catalog | VARCHAR | Name of the database. | | table\_schema | VARCHAR | Name of the schema. | | table\_name | VARCHAR | Name of the table. | | column\_name | VARCHAR | Name of the column. | | ordinal\_position | BIGINT | Ordinal of the column with respect to its table. | | data\_type | VARCHAR | Data type of the column (INT, CHAR, BOOLEAN, etc.). | | is\_nullable | VARCHAR | Specifies whether the values in this column can be NULL. | | column\_default | VARCHAR | The default expression of this column, if it exists. | #### information\_schema.data\_types The `data_types` view shows all data types in the system. | Column Name | Column Type | Column Description | | ----------- | ----------- | ------------------ | | data\_type | VARCHAR | Data type. | #### information\_schema.databases The `databases` view shows all databases where the current user has access. | Column Name | Column Type | Column Description | | -------------- | ----------- | ------------------------------------------------------- | | database\_name | VARCHAR | Name of the database. | | created\_at | TIMESTAMP | Timestamp that specifies when the database was created. | #### information\_schema.geometry\_columns The `geometry_columns` view shows information about all geometry type columns in the system. | Column Name | Column Type | Column Description | | ------------------- | ----------- | ----------------------------------------------------------- | | f\_table\_catalog | VARCHAR | Name of the database. | | f\_table\_schema | VARCHAR | Name of the schema. | | f\_table\_name | VARCHAR | Name of the table. | | f\_geometry\_column | VARCHAR | Name of the column. | | coord\_dimension | INT | The coordinate dimension. | | srid | INT | The ID of the spatial reference system. | | type | VARCHAR | Data type of the column (POINT, LINESTRING, POLYGON, etc.). | #### information\_schema.groups The `groups` view shows all groups in the system. | Column Name | Column Type | Column Description | | -------------- | ----------- | --------------------- | | database\_name | VARCHAR | Name of the database. | | group\_name | VARCHAR | Name of the group. | #### information\_schema.index\_recommendations This view contains index recommendations. | Column Name | Column Type | Column Description | | ------------ | ----------- | ------------------------------------------------------ | | table\_name | VARCHAR | Name of the table. | | column\_name | VARCHAR | Name of the column. | | sql | VARCHAR | SQL statement to create or drop the recommended index. | #### information\_schema.indexes The `indexes` view shows all indexes used in the system. | Column Name | Column Type | Column Description | | -------------- | ----------- | --------------------- | | table\_catalog | VARCHAR | Name of the database. | | table\_schema | VARCHAR | Name of the schema. | | table\_name | VARCHAR | Name of the table. | | index\_name | VARCHAR | Name of the index. | | index\_type | VARCHAR | Type of the index. | | column\_name | VARCHAR | Name of the column. | #### information\_schema.information\_schema\_catalog\_name The `information_schema_catalog_name` view shows the current database. | Column Name | Column Type | Column Description | | ------------- | ----------- | --------------------- | | catalog\_name | VARCHAR | Name of the database. | #### information\_schema.nodes The `nodes` view shows all nodes defined in the system. | Column Name | Column Type | Column Description | | -------------------- | ----------- | ---------------------------------------------------------------------------------------------------------- | | name | VARCHAR | Name of the node. | | service\_role\_types | VARCHAR\[] | The type of service roles on this node. | | operational\_status | VARCHAR | The operational status of the node. Values are ACTIVE, STARTING, STOPPING, ERROR, UNKNOWN, or UNREACHABLE. | | software\_version | VARCHAR | Version of the software running on this node. | #### information\_schema.pipeline\_status To view dynamic information about the status of the pipeline, query from information\_schema.pipeline\_status, or you can use the SHOW PIPELINE\_STATUS command. | Column Name | Column Type | Column Description | | ------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | database\_name | VARCHAR | Name of the database where the pipeline was created. | | pipeline\_name | VARCHAR | The name of this pipeline. | | table\_names | VARCHAR\[] | An array of fully-qualified table names where the pipeline loads data. | | status | VARCHAR | Status of the pipeline (RUNNING, STOPPED, COMPLETED, FAILED). | | status\_message | VARCHAR | The corresponding event message of the last event that the Ocient System sees for this pipeline from the event\_message column of the sys.pipeline\_events system catalog table. | | duration\_seconds | INT | Duration, in seconds, of how long the pipeline has been running. | | files\_processed | BIGINT | The number of files that have been processed for file-based loads. (includes files with these statuses: LOADED, LOADED\_WITH\_ERRORS, and SKIPPED) | | files\_failed | BIGINT | The number of files that have failed and have a file status of FAILED for file-based loads. | | files\_remaining | BIGINT | The number of remaining files for file-based loads. (includes these file statuses: PENDING, QUEUED, and LOADING) | | files\_total | BIGINT | The total number of files for file-based loads. | | fraction\_complete | FLOAT | The estimated fraction completion as a value from 0.0 to 1.0 for file-based batch or transactional loads. The calculation is files processed or failed divided by the total files. | | records\_processed | BIGINT | The number of records that the pipeline has processed. The pipeline attempts to process and load a record. This number can be greater than the value of records\_loaded due to the deduplication of records that the system processes twice or where record-level errors occur. | | records\_loaded | BIGINT | The number of records that have successfully loaded into the system. This number reflects the records in the system and does not include records that failed to load or were eliminated as duplicates. | | records\_failed | BIGINT | Number of records that have failed. | #### information\_schema.pipeline\_status\_historical To view information about the status of dropped pipelines, query from information\_schema.pipeline\_status\_historical. This system view is the historical equivalent of information\_schema.pipeline\_status. | Column Name | Column Type | Column Description | | ------------------ | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | database\_name | VARCHAR | Name of the database where the pipeline was created. | | pipeline\_name | VARCHAR | The name of this pipeline. | | status | VARCHAR | Status of the pipeline (RUNNING, STOPPED, COMPLETED, FAILED). | | status\_message | VARCHAR | The corresponding event message of the last event that the Ocient System sees for this pipeline from the `event_message` column of the `sys.pipeline_events` system catalog table. | | duration\_seconds | INT | Duration, in seconds, of how long the pipeline ran. | | files\_processed | BIGINT | The number of files that have been processed for file-based loads. (includes files with these statuses: LOADED, LOADED\_WITH\_ERRORS, and SKIPPED) | | files\_failed | BIGINT | The number of files that have failed and have a file status of FAILED for file-based loads. | | files\_remaining | BIGINT | The number of remaining files for file-based loads. (includes PENDING, QUEUED, and LOADING status files) | | files\_total | BIGINT | The total number of files for file-based loads. | | fraction\_complete | FLOAT | The estimated fraction completion as a value from 0.0 to 1.0 for file-based batch or transactional loads. The calculation is files processed or failed divided by the total files. | | records\_processed | BIGINT | The number of records that the pipeline has processed. | | records\_loaded | BIGINT | The number of records that have successfully loaded into the system. | | records\_failed | BIGINT | Number of records that have failed. | | pipeline\_id | UUID | Universally Unique IDentifier (UUID) of the pipeline. | #### information\_schema.pipeline\_table\_metrics To view dynamic information about the metrics specific to each of the target tables of the pipeline, query from the information\_schema.pipeline\_table\_metrics view. | Column Name | Column Type | Column Description | | -------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | database\_name | VARCHAR | The name of the database where the pipeline was created. | | pipeline\_name | VARCHAR | The name of this pipeline. | | table\_name | VARCHAR | The fully-qualified name of the table where the pipeline loads data. | | records\_processed | BIGINT | The number of records that the pipeline has processed for loading into the table. The pipeline attempts to process and load a record. This number can be greater than the value of the records\_loaded column due to the deduplication of records that the system processes twice or where the record-level errors occur. | | records\_loaded | BIGINT | The number of records the Ocient System has successfully loaded into the table. This number reflects the records in the table and does not include records that the pipeline failed to load or eliminated as duplicates. | | records\_failed | BIGINT | The number of records the pipeline failed to load into the table. This number does not include records the pipeline failed to load to all tables, such as failed records during extraction. | | processing\_duration | BIGINT | Duration, in milliseconds, indicating how long the pipeline has been processing records for the table. | | loading\_duration | BIGINT | Duration, in milliseconds, indicating how long the pipeline has been loading records into the table. | | bytes\_processed | BIGINT | Estimate of the amount of source data, in bytes, that the pipeline has processed for the table. | #### information\_schema.pipelines To view static information that stays the same after the pipeline has been created, query from information\_schema.pipelines, or you can use the SHOW PIPELINES command. The columns in the information\_schema.pipelines view derive from the underlying sys.pipelines system catalog table. | Column Name | Column Type | Column Description | | --------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------- | | database\_name | VARCHAR | Name of the database where the pipeline was created. | | pipeline\_name | VARCHAR | Name of the pipeline. | | loading\_mode | VARCHAR | Specifies whether the pipeline runs in a one-time (BATCH), one-time transactional (TRANSACTIONAL), or continuous (CONTINUOUS) way. | | source\_type | VARCHAR | Source of the data (S3, KAFKA, FILESYSTEM). | | status | VARCHAR | Status of the pipeline (RUNNING, STOPPED, COMPLETED, FAILED). | | data\_format | VARCHAR | Structure of the data (DELIMITED, CSV, JSON, PARQUET). | | table\_names | VARCHAR\[] | An array of fully-qualified table names where the pipeline loads data. | | created\_at | TIMESTAMP | Timestamp of when the pipeline was created. | | altered\_at | TIMESTAMP | Timestamp that indicates when the pipeline was last altered. | | creator\_id | UUID | Universally Unique IDentifier (UUID) of the creator of the pipeline. | | transaction\_id | UUID | The UUID of the transaction scope if this pipeline runs in a one-time transactional way. | #### information\_schema.pipelines\_historical To view static information about pipelines that have been dropped from the system, query from the `information_schema.pipelines_historical` system view. This view is the historical equivalent of the `information_schema.pipelines` view. | Column Name | Column Type | Column Description | | ------------------ | ----------- | ---------------------------------------------------------------------------------------------------------------------------------- | | database\_name | VARCHAR | Name of the database where the pipeline was created. | | pipeline\_name | VARCHAR | The name of this pipeline. | | loading\_mode | VARCHAR | Specifies whether the pipeline runs in a one-time (BATCH), one-time transactional (TRANSACTIONAL), or continuous (CONTINUOUS) way. | | source\_type | VARCHAR | Source of the data (S3, KAFKA, FILESYSTEM). | | status | VARCHAR | Status of the pipeline (RUNNING, STOPPED, COMPLETED, FAILED). | | data\_format | VARCHAR | Structure of the data (CSV, DELIMITED, JSON, BINARY, PARQUET, XML, ASN1, AVRO). | | created\_at | TIMESTAMP | Timestamp that indicates when this pipeline was created. | | altered\_at | TIMESTAMP | Timestamp that indicates when this pipeline was last updated. | | creator\_id | UUID | The Universally Unique IDentifier (UUID) of the user or group who created the pipeline (sys.users/sys.groups). | | transaction\_id | UUID | The UUID of the transaction scope if this pipeline runs in a one-time transactional way. | | dropped\_at | TIMESTAMP | Timestamp that indicates when the pipeline was dropped. | | rolehostd\_version | VARCHAR | The rolehostd version in which this pipeline was created. | | pipeline\_id | UUID | UUID of the pipeline. | #### information\_schema.reserved\_words The `reserved_words` view shows all reserved words in the system. | Column Name | Column Type | Column Description | | -------------- | ----------- | ---------------------------------------------------- | | reserved\_word | VARCHAR | A word that is a reserved keyword for use by Ocient. | #### information\_schema.schemata The `schemata` view shows information for all schemas in the system. | Column Name | Column Type | Column Description | | ------------- | ----------- | --------------------- | | schema\_name | VARCHAR | Name of the schema. | | catalog\_name | VARCHAR | Name of the database. | #### information\_schema.table\_privileges The `table_privileges` view shows information for all table privileges in the system. | Column Name | Column Type | Column Description | | --------------- | ----------- | ---------------------------------------------------------- | | grantor | VARCHAR | The user who granted this privilege. | | grantee | VARCHAR | The user who received this privilege. | | table\_catalog | VARCHAR | Name of the database. | | table\_schema | VARCHAR | Name of the schema. | | table\_name | VARCHAR | Name of the table. | | privilege\_type | VARCHAR | The privilege for the grant. | | is\_grantable | VARCHAR | Whether the user can grant this privilege to another user. | #### information\_schema.table\_storage The `tables_storage` view shows information about the storage used by tables. | Column Name | Column Type | Column Description | | ------------------- | ----------- | ------------------------------------------------- | | table\_catalog | VARCHAR | Name of the database. | | table\_schema | VARCHAR | Name of the schema. | | table\_name | VARCHAR | Name of the table. | | row\_count | BIGINT | The number of rows in the table. | | deleted\_row\_count | BIGINT | The number of deleted rows in the table. | | segment\_count | BIGINT | The number of segments that constitute the table. | | size | BIGINT | The size of the table in bytes. | #### information\_schema.tables The `tables` view shows information for all tables and views in the system, including user-defined and system objects. | Column Name | Column Type | Column Description | | -------------------- | ----------- | ----------------------------------------------------------------- | | table\_catalog | VARCHAR | Name of the database. | | table\_schema | VARCHAR | Name of the schema. | | table\_name | VARCHAR | Name of the table. | | table\_type | VARCHAR | Type of the table. | | is\_insertable\_into | VARCHAR | Whether an INSERT command can target this table. | | created\_at | TIMESTAMP | Timestamp that represents when the table was created. | | creator\_id | UUID | Universally Unique IDentifier (UUID) of the creator of the table. | #### information\_schema.transactional\_pipeline\_status To view information about the status of transactional pipelines (both active and dropped), query from information\_schema.transactional\_pipeline\_status. This system view combines information\_schema.pipeline\_status and information\_schema.pipeline\_status\_historical, filtered to pipelines with a non-NULL `transaction_id` column. | Column Name | Column Type | Column Description | | ------------------ | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | transaction\_id | UUID | The Universally Unique IDentifier (UUID) of the transaction scope for this pipeline. | | database\_name | VARCHAR | Name of the database where the pipeline was created. | | pipeline\_name | VARCHAR | The name of this pipeline. | | table\_names | VARCHAR\[] | An array of fully-qualified table names where the pipeline loads data. This value is NULL for dropped pipelines. | | status | VARCHAR | Status of the pipeline (RUNNING, STOPPED, COMPLETED, FAILED). | | status\_message | VARCHAR | The corresponding event message of the last event that the Ocient System sees for this pipeline from the `event_message` column of the `sys.pipeline_events` system catalog table. | | duration\_seconds | INT | Duration, in seconds, of how long the pipeline has been running. | | files\_processed | BIGINT | The number of files that have been processed for file-based loads. (includes files with these statuses: LOADED, LOADED\_WITH\_ERRORS, and SKIPPED) | | files\_failed | BIGINT | The number of files that have failed and have a file status of FAILED for file-based loads. | | files\_remaining | BIGINT | The number of remaining files for file-based loads. (includes PENDING, QUEUED, and LOADING status files) | | files\_total | BIGINT | The total number of files for file-based loads. | | fraction\_complete | FLOAT | The estimated fraction completion as a value from 0.0 to 1.0 for file-based batch or transactional loads. The calculation is files processed or failed divided by the total files. | | records\_processed | BIGINT | The number of records that the pipeline has processed. | | records\_loaded | BIGINT | The number of records that have successfully loaded into the system. | | records\_failed | BIGINT | Number of records that have failed. | | pipeline\_id | UUID | UUID of the pipeline. | #### information\_schema.users The `users` view shows all users in the system. | Column Name | Column Type | Column Description | | -------------- | ----------- | --------------------- | | database\_name | VARCHAR | Name of the database. | | user\_name | VARCHAR | Username of the user. | #### information\_schema.views The `views` view shows information for all user-defined and system views in the system. | Column Name | Column Type | Column Description | | ---------------- | ----------- | ---------------------------------------------------------------------------- | | table\_catalog | VARCHAR | Name of the database. | | table\_schema | VARCHAR | Name of the schema. | | table\_name | VARCHAR | Name of the table. | | view\_definition | VARCHAR | Query used to generate the view content. | | created\_at | TIMESTAMP | Timestamp that represents the date and time for the creation of the view. | | updated\_at | TIMESTAMP | Timestamp that represents the date and time for the last update of the view. | | creator\_id | UUID | Universally Unique IDentifier (UUID) of the creator of the view. | ## SHOW Commands The `SHOW` commands are additional commands you can use to retrieve metadata about the system. These commands are tightly coupled with the `information_schema` schema and offer a more user-friendly syntax alternative. Sometimes, unlike their `information_schema` counterparts, `SHOW` commands only return user-defined objects. | SHOW Command | Description | | ---------------------- | -------------------------------------------------------------------------------------------------------- | | SHOW COLUMNS | This statement returns columns from user-defined tables and views where the current user has access. | | SHOW DATA\_TYPES | This statement returns all data types in the system. | | SHOW DATABASES | This statement returns all databases where the current user has access. | | SHOW GEOMETRY\_COLUMNS | This statement returns all geometry type columns where the current user has access. | | SHOW GROUPS | This statement returns all groups in the system where the current user has read access. | | SHOW INDEXES | This statement returns all indexes on user-defined tables where the current user has access. | | SHOW NODES | This statement returns all nodes in the system where the current user has read access. | | SHOW PIPELINE\_STATUS | This statement returns the status of all pipelines in the system where the current user has read access. | | SHOW PIPELINES | This statement returns all pipelines in the system where the current user has read access. | | SHOW RESERVED\_WORDS | This statement returns all reserved words in the system. | | SHOW SCHEMATA | This statement returns all user-defined schemas where the current user has access. | | SHOW SYSTEM TABLES | This statement returns all system tables in the `sys` or `information_schema` schemas. | | SHOW TABLE\_PRIVILEGES | This statement returns all table privileges for tables where the current user has access. | | SHOW TABLES | This statement returns all user-defined tables where the current user has access. | | SHOW USERS | This statement returns all users in the system where the current user has read access. | | SHOW VIEWS | This statement returns all user-defined views where the current user has access. | ## Related Links [SQL Reference](/sql-reference) [Database Administration](/database-administration) [System Catalog](/system-catalog) # Ingest Data with Legacy LAT Reference Source: https://docs.ocient.com/ingest-data-with-legacy-lat-reference Explore methods for efficiently loading large data sets into Ocient, optimized for performance and designed for massive-scale data warehousing. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). ## Loading and Transformation (LAT) Overview This section provides an overview of how loading works in Ocient along with step-by-step examples to illustrate the key aspects of the loading process. A separate LAT Reference provides a detailed explanation of the options for each data format and data source in Ocient. The key elements in the Loading system are: * Data Source — an origin source for data such as S3, , or . Sources can be file or streaming in nature. * Data Type Extraction — the format for data extraction from the source. Examples include JSON, CSV, fixed width. * Transformations — the functions used to cleanse, route, and transform the incoming data. * Indexer — operated by the `streamloader` role on a Loader Node, the Indexer collects transformed records, stores in replicated pages, and converts into segments on the Foundation Nodes. Reference: * [LAT Data Types in Loading](/lat-data-types-in-loading) * [LAT Pipeline Configuration](/lat-pipeline-configuration) * [LAT Client Command Line Interface](/lat-client-command-line-interface) * [LAT Metrics](/lat-metrics) * [LAT Advanced Topics](/lat-advanced-topics) * [LAT Packaging and Installation](/lat-packaging-and-installation) ### Loading Overview The LAT is a service within an Ocient System that is responsible for fetching data from file or streaming sources, extracting records, transforming them, and routing them to Ocient tables. The LAT runs as a standalone process on Loader Nodes. It runs many parallel workers on high-core count servers to deliver the required parallelism for high-throughput loading. This diagram shows that a set of Loader Nodes sit between the data sources and the Foundation Nodes to control the loading flow and deliver high performance across many parallel workers. Loading and transformation into foundation storage The LAT loads data from two general types of data sources: * Streaming Sources — continuous loading of data from an ordered stream such as Kafka * File Sources — a discrete batch of data loaded from files until complete, typically from a file system or object storage solution like AWS S3 The main difference between these modes is whether the task continues indefinitely or ends when the batch of files has been loaded. In both cases, the LAT uses identical transformation and loading processes so that users can manage both with the same mental model and configuration files. Most data types are supported identically in streaming and batch processing modes, however some formats are not suitable for streaming due to their structure. A complete list of data sources and supported data formats is in [LAT Data Types in Loading](/lat-data-types-in-loading). While loading, the LAT coordinates with the Ocient System to manage backpressure of loading new records. New rows accumulate into a storage structure called a "page" which is a column-oriented storage mechanism. These pages are replicated across Foundation Nodes so that the specified redundancy is maintained in the event of a system outage. When enough pages have accumulated in a given time period (as defined by the bucket), the Indexer will convert the pages into segment structures and place these on the Foundation Nodes for permanent storage as Segment Groups that span the width of the storage space. This process is transparent to analysts issuing queries. Rows are federated during queries so that pages and Segments are seamlessly presented in result sets. ### Pipelines Overview The process and configuration that defines how data is loaded is referred to as a "pipeline." A "pipeline" defines the end-to-end data flow in a loading task including the Source type (e.g., Kafka), the record Extraction for the data type (e.g., CSV extraction), the transformations on each record (e.g., JSON value extraction, data exploding, string concatenation, flattening), and where transformed data should be loaded in Ocient tables. As shown in this diagram, a user sets each of these sections in the pipeline file to control the LAT. LAT pipeline JSON configuration with data flow from data sources to storage and processing The LAT pipelines are managed through an HTTP API on the Loader Nodes or through a Command Line Interface (CLI) that provides a convenient way to work with the HTTP API. Because most Ocient Systems include many Loader Nodes, the CLI coordinates pipelines across the specified set of Loader Nodes. The LAT Client issues the command to the Loader Node to create the new pipeline with the pipeline configuration file. When started, the LAT uses the pipeline to execute a highly parallelized loading task across all Loader Nodes and Foundation Nodes to load the configured tables. Detailed instructions for configuring a pipeline are found in [LAT Pipeline Configuration](/lat-pipeline-configuration). ### Time Ordering The Ocient System contains timeseries data. Loading in the data warehouse is considerably faster when data is presented in an ordered time sequence according to the TimeKey in a table. For most streaming sources, this is typically accomplished automatically with limited "out of order" data by the nature of the queue. However, for file based sources, it is not uncommon for data to be collected in folders in a manner that could appear haphazard to the loader. For this reason, it is critical that the `sort_type` setting for file based sources be used correctly to inform the LAT how files should be ordered. Read more about file sorting in [Load From a File Source](/lat-source-configuration#load-from-a-file-source). ### Exactly-Once Guarantees Loading in Ocient is designed to ensure exactly once processing of data. The LAT operates independent streams of data that each has a monotonically increasing row ID tied to an individual row in a File-based load or a partition offset in a Kafka-based load. Exactly-once processing is made possible through coordination between the loading components around a "Durability Horizon" that represents the highest record durably stored on non-volatile storage in each independent stream. In the event of a node outage or a replay of the data, the Durability Horizon automatically removes duplicate records in an efficient manner. It is important to understand how the LAT determines the unique row ID for different source types to ensure that data is loaded correctly. This is described in more detail in the LAT Reference Documentation. Learn more about the underlying approach used in [Loading Characteristics and Concepts](/loading-characteristics-and-concepts). ### Dynamic Schema Changes Ocient also supports schema changes on tables while loading data in a continuously streaming pipeline. Ocient maintains a table version with each running pipeline that serves as a loading contract until the load task complete. This ensures that existing pipelines are not interrupted when columns are added or removed from a table that is receiving new data. Ocient continues loading records using the original table version even after a change has been made to the tables. These dynamic schema changes from `ALTER TABLE` commands allow flexible updates to tables while complex loading processes are active. Pipelines can then be updated to add or remove data elements and match the altered schema, reducing the burden of coordination across systems. ### Loading Examples The following examples walk through a simple loading example using the LAT for a streaming load off of Kafka and a file load off of S3. These examples are very similar due to the way the LAT uses a common language for all transformation and loading regardless of data source or data type. The final set of examples outline some more complex transformations. Examples: * [LAT Load JSON Data from Kafka](/lat-load-json-data-from-kafka) * [LAT Load CSV Data from S3](/lat-load-csv-data-from-s3) * [LAT Load JSON Data from S3](/lat-load-json-data-from-s3) * [LAT Advanced Loading and Transformations](/lat-advanced-loading-and-transformations) For more detailed information about LAT settings, see the [LAT Pipeline Configuration](/lat-pipeline-configuration). ## Related Links [LAT Overview](/lat-overview) [LAT Data Types in Loading](/lat-data-types-in-loading) # Inspect the Current Configuration Source: https://docs.ocient.com/inspect-the-current-configuration Inspect the current configuration of an Ocient System using SQL queries, the REST API, and system catalog tables to verify settings before changes. includes a variety of configuration settings that control how the system behaves. The vast majority of these do not change in a typical deployment. However, there is a small set more commonly changed to adjust for different workloads, hardware, and operating requirements. Configuration is primarily done through DDL or DCL and can be inspected using the [System Catalog Reference](/system-catalog-reference). You can contact Ocient Support to change configuration settings that apply across the entire system or to a subset of nodes. Each Ocient node reads configuration settings during startup. Therefore, configuration changes require restarts for the new values to take effect. For details, contact Ocient Support. ## Inspecting System Configuration Administrators typically need to know the effective configuration of the system and what has been overridden. * Configuration Overrides — a list of the overrides applied by administrators * Effective Configuration — the complete list of all configuration parameters and their current values on each node ### Viewing Configuration Overrides A system catalog table is provided that lists all DDL Configuration Overrides that are applied to the Ocient System. This table includes the configuration target type as well as the scope to which it is applied. In the event that the scope is not `SYSTEM` (See Advanced Configuration for `CLUSTER` and `NODE` scopes), the `scope_id` indicates the `cluster_id` or `node_id` to which the configuration override applies. **Example** The following query retrieves all config overrides applied on the system and provides a friendly node name or cluster name where an override is limited in scope: ```sql SQL theme={null} SELECT cfg.scope_type, n.name AS node_name, c.name AS cluster_name, cfg.key, cfg.value FROM sys.config cfg LEFT JOIN sys.nodes n ON n.id = cfg.scope_id and cfg.scope_type = 'NODE' LEFT JOIN sys.clusters c ON c.id = cfg.scope_id and cfg.scope_type = 'CLUSTER'; ``` Output ```sql SQL theme={null} scope_type node_name cluster_name key value --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- NODE sales-lts8 NULL lts.memory.warningHeapMemoryUsageRatio 0.93999999999999995 SYSTEM NULL NULL lts.blobstoreParameters.coreMask 0b10000000000000000000 CLUSTER NULL storageCluster1 lssParameters.nvmeParameters.useOpal true NODE sales-lts8 NULL lts.lssParameters.nvmeParameters.ioCoreMask 68719542272 SYSTEM NULL NULL sql.pdfReservedMem 137438953472 ... ``` ### Viewing Effective Configuration Using System Table A system catalog table is provided that lists the effective configuration of each node. **Example** The following query retrieves the effective overrides for the SQL role on the specified node: ```sql SQL theme={null} SELECT n.name, nc.key, nc.value FROM sys.node_config nc JOIN sys.nodes n ON nc.node_id = n.id WHERE n.name = 'sql0' AND key LIKE 'sql%'; ``` Output ```none Text theme={null} name key value --------------------------------------------------------------------------------------------------------------------------------------- sql0 sql.blobstoreParameters.regionAllocationUnit 17179869184 sql0 sql.blobstoreParameters.pageUtilizationExpand 0.75 sql0 sql.blobstoreParameters.pageUtilizationContract 0.25 sql0 sql.blobstoreParameters.maxFileSize 0 sql0 sql.blobstoreParameters.minFileSize 128MiB sql0 sql.blobstoreParameters.blobPageSize 128KiB sql0 sql.blobstoreParameters.stripeWidth 0 sql0 sql.blobstoreParameters.coreMask 0b10000000 sql0 sql.cacheMinFreeMemoryPercent 0.109999999 sql0 sql.cacheMinSize 4096 sql0 sql.cacheMinKeepTime 720 sql0 sql.purgeCacheFrequency 300 sql0 sql.pdfReservedMem 0 sql0 sql.numLevels 3 sql0 sql.maxShutdownWaitTime 1800000 sql0 sql.shortQueryMillis -1 sql0 sql.timeLimit 1800000 sql0 sql.forceNetworkedVirtualTables false sql0 sql.cacheLimit 17179869184 sql0 sql.optimizer TKTOptimizer sql0 sql.generator TKTPlanGenerator sql0 sql.validator TKTValidator sql0 sql.concurrencyTarget 5 ``` ### Viewing Effective Configuration Using REST Endpoints Administrators can also view the effective system configuration using the config endpoint, a REST interface available on each node. This endpoint shows every configuration parameter for the node and its current value. It reflects the active values and might not reflect configuration overrides applied since the last node restart. The config endpoint is found on each node at `http://:9090/v1/config` and it returns a JSON representation of the active system configuration of the targeted node. **Example** ```curl CURL theme={null} curl http://10.0.1.7:9090/v1/config ``` Output ```json JSON theme={null} { "siloType": "xg::runtime::physicalSiloSet_t", "tlbfsInfo.enabled": "true", "gdsClientParameters_t.batchTimer": "150000000n", "gdsClientParameters_t.maxBatchCount": "4096" ... "operatorvm.vmSiloMask": "0x3", "operatorvm.vmParameters.levels": "0x4", "operatorvm.vmParameters.statsCacheSize": "4GiB", "operatorvm.vmProtocolParameters.routerPollRate": "250000000n", "operatorvm.vmProtocolParameters.bufferedDatablockTimeout": "120000000000n" ... "lts.ioOpParameters.bufferPool.backingSlabSize": "2MiB", "lts.ioOpParameters.bufferPool.bufferPools[0].pool.size": "4KiB", "lts.ioOpParameters.bufferPool.bufferPools[0].pool.count": "98304", "lts.ioOpParameters.lbaCacheSize": "64", "lts.targetFillRatio": "1", "lts.pdfReservedMem": "4GiB" ... } ``` ## Related Links [System Catalog](/system-catalog) # Install an Ocient System Source: https://docs.ocient.com/install-an-ocient-system Install an Ocient System with this end-to-end guide covering hardware setup, OS preparation, application installation, node bootstrapping, and validation. This section provides a quick tutorial for installing and configuring the System using your data. ## Installation of the Ocient System Before installing, check with your systems administrator to ensure that you meet the [Ocient System Requirements](/ocient-system-requirements) for the validated operating systems and hardware needed for optimized Ocient performance. The installation and configuration of the Ocient System involves these steps: [Ocient Application Installation](/ocient-application-installation) This step details how to download and install the Ocient System package. [Ocient System Bootstrapping](/ocient-system-bootstrapping) This step describes the Ocient bootstrapping, which sets up the initial node configuration and connects the nodes together. [Ocient Application Configuration](/ocient-application-configuration) This step sets up your Ocient System with storage spaces, storage clusters, and node roles. These pages provide additional information, validation tools, and configuration options for installing your Ocient® System. * [**Ocient System Sizing**](/ocient-system-sizing) — Guidance on the factors that influence Ocient System sizing. * [**Ocient System Requirements**](/ocient-system-requirements) — Information on hardware and system requirements for optimized Ocient performance. * [**Ocient System Hardware Specifications**](/ocient-system-hardware-specifications) — Information on the detailed hardware specification for the Ocient System. * [**Operating System Configuration**](/operating-system-configuration) — A tutorial to configure the necessary prerequisites for system bootstrapping. * [**Checking System Configuration Before Bootstrapping**](/checking-system-configuration-before-bootstrapping) — Commands to capture critical node health states and metrics before beginning the bootstrapping process. * [**Checking System After Configuration**](/checking-system-after-configuration) — Commands to capture critical node health states and metrics after the final configuration process. * [**Node Bootstrapping Reference**](/node-bootstrapping-reference) — Descriptions of additional options for advanced system configurations. # Install LAT Client in a Disconnected Network Source: https://docs.ocient.com/install-lat-client-in-a-disconnected-network Install the Ocient Loading and Transformation (LAT) client in a disconnected or air-gapped network, including offline package distribution and setup steps. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). Operate the LAT Client in a disconnected network using these steps to install the LAT Client. ## Requirements On the version 8.0, the version 3.6.8 specifies the default `python3` requirement and the LAT Client requires Python version 3.8 or greater. Contact Ocient Support for the LAT Client Python wheel file. Install the LAT Client Python wheel by first gathering the RPMs for Python 3.11 on a system with access to the appropriate software repository `python3.11` and its dependencies: * `mpdecimal` * `python3.11-libs` * `python3.11-pip-wheel` * `python3.11-setuptools-wheel` Use your regular path for the Python packages. Gather the `lat_client` and its dependencies (`.whl` files). Execute this download command on a machine with Python 3.11 and `pip` installed. ```shell Shell theme={null} $ pip download lat_client-3.1.2-py3-none-any.whl ``` This command downloads these dependencies. ```shell Shell theme={null} lat_client aiohttp==3.9.3 aiosignal==1.3.1 frozenlist==1.4.1 attrs==23.2.0 frozenlist==1.4.1 multidict==6.0.5 yarl==1.9.4 idna==3.6 multidict==6.0.5 more-itertools==10.2.0 pydantic==2.6.4 annotated-types==0.6.0 pydantic_core==2.16.3 typing_extensions==4.10.0 typing_extensions==4.10.0 rich==13.7.1 markdown-it-py==3.0.0 mdurl==0.1.2 Pygments==2.17.2 wheel==0.43.0 ``` Copy the RPMs and wheel files to the machine in the disconnected network. ## Install the LAT Client Install Python 3.11 on the machine in the disconnected network where `` is the path to the Python 3.11 RPM. `` is the path to a dependency of Python 3.11. You can list more dependencies in this way using spaces between each one. ```shell Shell theme={null} dnf install ``` After Python 3.11 is installed, create a `venv` to install the LAT Client and its dependencies. ```shell Shell theme={null} $ /usr/bin/python3.11 -m venv venv ``` Enter the `venv`. ```shell Shell theme={null} $ source venv/bin/activate ``` Install `lat_client` and its dependencies in the `venv`. Specify wheel files `` as paths to wheel files separated by spaces. ```shell Shell theme={null} (venv)$ pip install lat_client-3.1.2-py3-none-any.whl ``` Check that the LAT Client runs. ```shell Shell theme={null} (venv)$ lat_client --version 3.1.2 ``` ## Related Links [LAT Client Command Line Interface](/lat-client-command-line-interface) # Installation Reference Source: https://docs.ocient.com/installation-reference Reference for installing an Ocient System, including supported platforms, hardware checks, package layout, configuration files, and post-install validation. These pages provide additional information, validation tools, and configuration options for installing your Ocient® System. * [**Ocient System Sizing**](/ocient-system-sizing) — Guidance on the factors that influence Ocient System sizing. * [**Ocient System Requirements**](/ocient-system-requirements) — Information on hardware and system requirements for optimized Ocient performance. * [**Ocient System Hardware Specifications**](/ocient-system-hardware-specifications) — Information on the detailed hardware specification for the Ocient System. * [**Operating System Configuration**](/operating-system-configuration) — A tutorial to configure the necessary prerequisites for system bootstrapping. * [**Checking System Configuration Before Bootstrapping**](/checking-system-configuration-before-bootstrapping) — Commands to capture critical node health states and metrics before beginning the bootstrapping process. * [**Checking System After Configuration**](/checking-system-after-configuration) — Commands to capture critical node health states and metrics after the final configuration process. * [**Node Bootstrapping Reference**](/node-bootstrapping-reference) — Descriptions of additional options for advanced system configurations. # JDBC Classes and Methods Source: https://docs.ocient.com/jdbc-classes-and-methods Reference of JDBC classes and methods supported by the Ocient JDBC driver, including Connection, Statement, PreparedStatement, ResultSet, and metadata APIs. This page lists methods that are currently not supported by the JDBC Driver for use in Java programs. The Ocient JDBC driver supports methods not listed on this page. See the linked reference Java 8 documentation pages for details on the functionality of each class. | **Java Class** | **Unsupported Methods or Comments** | | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [DriverManager](https://docs.oracle.com/javase/8/docs/api/java/sql/DriverManager.html) | There are no unsupported methods.

Get a connection with getConnection().
getConnection(String, Properties)
The string is a URL and must be of the form:
`jdbc:ocient://IP:Port/databaseName[;property=value;…]`

For description of supported properties, see [JDBC Connection Properties](/jdbc-manual#supported-jdbc-connection-properties). | | [Connection](https://docs.oracle.com/javase/8/docs/api/java/sql/Connection.html) | createSQLXML() prepareCall() releaseSavepoint() rollback() setNetworkTimeout() setSavepoint() | | [Statement](https://docs.oracle.com/javase/8/docs/api/java/sql/Statement.html) | clearBatch() closeOnCompletion() getGeneratedKeys() getMoreResults() setCursorName() setEscapeProcessing() setMaxFieldSize() setPoolable() | | [PreparedStatement](https://docs.oracle.com/javase/8/docs/api/java/sql/PreparedStatement.html) | getParameterMetaData() setRef() setRowId() setSQLXML() setUnicodeStream() setURL() | | [ResultSet](https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSet.html) | absolute() afterLast() beforeFirst() cancelRowUpdates() deleteRow() first() getCursorName() getRef() getRowId() getSQLXML() getUnicodeStream() getURL() insertRow() isLast() last() moveToCurrentRow() moveToInsertRow() previous() refreshRow() relative() updateArray() updateAsciiStream() updateBigDecimal() updateBinaryStream() updateBlob() updateBoolean() updateByte() updateBytes() updateCharacterStream() updateClob() updateDate() updateDouble() updateFloat() updateInt() updateLong() updateNCharacterStream() updateNClob() updateNString() updateNull() updateObject() updateRef() updateRow() updateRowId() updateShort() updateSQLXML() updateString() updateTime() updateTimestamp() | | [ResultSetMetaData](https://docs.oracle.com/javase/8/docs/api/java/sql/ResultSetMetaData.html) | All methods are supported. | | [DatabaseMetaData](https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html) | All methods are supported. | ## Related Links [Connect Using JDBC](/connect-using-jdbc) # JDBC Manual Source: https://docs.ocient.com/jdbc-manual Manual for the Ocient JDBC driver, including installation, connection URLs, supported properties, authentication, query execution, and result-set handling. The JDBC Driver and command-line interface (CLI) enable you to connect to Ocient using a JDBC connection. Ensure that you meet the prerequisites before using the Ocient JDBC Driver. Then, invoke the CLI program, configure options, and connect to a database using the driver. You can also use the data extract tool to extract a result set to delimited files in the target location. For details about data extracting, see [Data Extract Tool](/data-extract-tool). For a list of commands available in the JDBC CLI, see [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). ### Prerequisites This software is required for the JDBC driver. | **Software** | **Version** | | -------------------------------------------------------- | ------------------------------------------------ | | Ocient | Use the latest Ocient System version. | | Operating System (OS) | , , or .
Use the latest version of each OS. | | See the [Version Compatibility](/version-compatibility). | | ### Driver Features The Ocient JDBC connector supports these features as of the current version. | Unicode Support | UTF-8 | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Isolation Levels | Ocient does not support transactions at this time. | | Data Types | Supports all Ocient [Data Types](/data-types). | | Security / Encryption | Uses SSL/TLS to connect to the Ocient System.
TLS protocol is available as a [JDBC Configuration Option](#jdbc-cli-configuration-options). | ### Invoke the Ocient JDBC CLI Program If your system meets all the necessary prerequisites, you can run the JDBC CLI by using the Ocient JDBC JAR file. To do this, follow these steps. Go to the Ocient [repository](https://mvnrepository.com/artifact/com.ocient/ocient-jdbc4) for all JDBC versions. For more information on which version to pick, see the [Version Compatibility](/version-compatibility) page. For the JDBC version you want to use, download the JAR file with dependencies. This JAR file follows the format: `ocient-jdbc4--jar-with-dependencies.jar`. Move this JAR file to the directory where your Ocient System is installed. From the shell terminal, run this command to launch the JDBC CLI. ```shell Shell theme={null} java -classpath com.ocient.cli.CLI [ []] ``` This example runs JDBC version 2.10. ```shell Shell theme={null} java -classpath ./ocient-jdbc4-2.10-jar-with-dependencies.jar com.ocient.cli.CLI testuser testpassword ``` After launching, the JDBC CLI prompts you to enter your username and password. ```shell Shell theme={null} Username: admin@system Password: admin ``` The interface changes to the Ocient CLI. ```shell Shell theme={null} Ocient> _ ``` Connect to your system from the JDBC using a connection string. Assuming the standard port `4050`, a self-signed certificate, and the SQL Node IP address `10.10.1.1`, you can connect to the system database with the following connecting string. ```shell Shell theme={null} Ocient> connect to jdbc:ocient://10.10.1.1:4050/system; ``` The CLI responds with a connection message. ```shell Shell theme={null} Connected to jdbc:ocient://10.10.1.1:4050/system Ocient> _ ``` Now that you are connected to your system, you can execute any queries or commands. For Java version 1.8.0\_144, [download](https://www.oracle.com/java/technologies/javase-jce8-downloads.html) and install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 8. #### JDBC CLI Configuration Options The JDBC CLI reads a configuration file consisting of key-value pairs located at `~/.ocient-cli-configuration` with this format. ```shell Shell theme={null} key1=value1 key2=value2 ... keyN=valueN ``` The JDBC CLI supports these options. | **Option** | **Description** | **Default Value** | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | `cliIdleTimeoutMinutes` | Configures the idle timeout for the CLI. The CLI rejects subsequent commands and exits the process after `cliIdleTimeoutMinutes` minutes of inactivity. | `0` if left unspecified or the configuration file does not exist. | | `performance` | Enables different levels of query result output for measuring benchmarks by using different performance options. For details, see the [PERFORMANCE](/commands-supported-by-the-ocient-jdbc-cli-program#performance) command. | `off` | | `printUuid` | Accepts values `on` or `off`.
If you set the `printUuid` option to `on`, the system prints the query identifier for each query that you execute in the CLI. | `off` | | `timing` | Enables or disables reporting the execution time of each query. For details, see the [TIMING](/commands-supported-by-the-ocient-jdbc-cli-program#timing) command. | `off` | For supported commands, see [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). #### Run a Transaction in the JDBC CLI The Ocient JDBC CLI supports multi-statement transactions using SQL statements. To begin a transaction, execute the `SET AUTOCOMMIT OFF` SQL statement, and then end the transaction with the `COMMIT` or `ROLLBACK` statements. A query within the transaction reads uncommitted rows that the same connection has inserted. This example disables autocommit, inserts two rows into the `public.txn_demo` table, counts the rows in the table, and commits the transaction. Next, the example inserts two more rows in the table and counts the rows. Then, the example runs a `ROLLBACK` command to roll back the changes, and counts the rows again. ```sql SQL theme={null} Ocient> SET AUTOCOMMIT OFF; Ocient> INSERT INTO public.txn_demo VALUES (1, 'alpha'); Ocient> INSERT INTO public.txn_demo VALUES (2, 'beta'); Ocient> SELECT count(*) AS n_in_txn FROM public.txn_demo; Ocient> COMMIT; Ocient> INSERT INTO public.txn_demo VALUES (3, 'gamma'); Ocient> INSERT INTO public.txn_demo VALUES (4, 'delta'); Ocient> SELECT count(*) AS n_in_txn FROM public.txn_demo; Ocient> ROLLBACK; Ocient> SELECT count(*) FROM public.txn_demo; ``` ### Use the Ocient JDBC Driver in Java Programs First, you must load the Ocient driver class with this statement in a Java program using the JDBC driver. ```java Java theme={null} Class.forName("com.ocient.jdbc.JDBCDriver"); ``` The driver class is located in the JDBC driver JAR file named `ocient-jdbc4.jar` and must be available in the CLASSPATH defined for the program. ### Connect Using JDBC The Ocient JDBC driver supports connection properties that can be supplied using the [CONNECT](/commands-supported-by-the-ocient-jdbc-cli-program#connect) command in the JDBC CLI or as a properties object passed in a Java application. **JDBC URL Example** This connection string includes various connection properties that trail the login credentials. ```shell Shell theme={null} CONNECT TO 'jdbc:ocient://db.example.com:6432/salesdb;user=admin@system;password=admin;logLevel=INFO;networkTimeout=15000;enableBulkLoad=true;bulkLoadThreshold=50000'; ``` **Java DriverManager Example** Alternatively, you can use the DriverManager class to provide connection properties if you are using a JDBC connection as part of a Java application. ```java Java theme={null} import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.util.Properties; public class OcientConnectionExample { public static void main(String[] args) throws SQLException { // Base JDBC URL: host, port, and database String url = "jdbc:ocient://db.example.com:6432/salesdb"; // JDBC connection properties Properties props = new Properties(); props.setProperty("user", "analytics_user"); // Ocient username props.setProperty("password", "StrongPassw0rd!"); // Ocient password props.setProperty("logLevel", "INFO"); // Driver logging verbosity props.setProperty("networkTimeout", "15000"); // 15s network timeout (ms) props.setProperty("enableBulkLoad", "true"); // Use bulk load for large batches props.setProperty("bulkLoadThreshold", "50000"); // Min rows for bulk loa try (Connection conn = DriverManager.getConnection(url, props)) { } } } ``` #### Supported JDBC Connection Properties The JDBC driver supports these connection parameters. | **Parameter** | **Description** | | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bulkLoadChunkSize` | The number of rows to include in each JSON data file (chunk) uploaded. The default value is `60000`. | | `bulkLoadCleanupOnFailure` | If you set this parameter value to `true`, the driver deletes the temporary files and pipelines even if the load fails. The default value is `true`. | | `bulkLoadFailOnError` | If you set this parameter value to `true`, `executeBatch()` fails if any `INSERT` operation fails.

If you set this parameter value to `false`, `executeBatch()` falls back to a standard multi-row `INSERT` operation if any INSERT operation fails. The default value is `false`. | | `bulkLoadLoaderNodesCacheSeconds` | The number of seconds to cache the list of active Loader Nodes the system discovers during bulk load. When you set this parameter to a value greater than `0`, the driver caches the Loader Node list for the specified duration and forwards the same value as a server-side `CACHE_MAX_TIME` hint on the lookup query. This cache is -wide and is not specific to individual connections. Do not enable this parameter if your application connects to multiple Ocient clusters from the same JVM. The default value is `0` (disabled). | | `bulkLoadMode` | The bulk load transport mode. Set this parameter to `ssh` to stage data using SSH/SFTP to Loader Nodes, or set it to `s3` to stage data in an S3-compatible object store. The default value is `ssh`. | | `bulkLoadPollIntervalSeconds` | The number of seconds to wait between polling the `sys.pipelines` system catalog table for load status. The default value is `2`. | | `bulkLoadS3AccessKeyId` | The access key identifier for authenticating to the S3 endpoint. This parameter is required when you set the `bulkLoadMode` parameter to `s3`. | | `bulkLoadS3ApiCallTimeoutMs` | The timeout, in milliseconds, for individual S3 API calls. The default value is `120000` (two minutes). | | `bulkLoadS3Bucket` | The S3 bucket name for staging bulk load data files. This parameter is required when you set the `bulkLoadMode` parameter to `s3`. | | `bulkLoadS3EnablePathStyleAccess` | If you set this parameter to `true`, the driver uses path-style access for S3 requests (e.g., `http://endpoint/bucket/key` instead of `http://bucket.endpoint/key`). This parameter is required for S3-compatible services such as MinIO or Ceph. The default value is `true`. | | `bulkLoadS3Endpoint` | The S3-compatible endpoint URL for bulk load staging (e.g., `https://s3.us-east-1.amazonaws.com`). This parameter is required when you set the `bulkLoadMode` parameter to `s3`. | | `bulkLoadS3MultipartPartSize` | The size, in bytes, of each part in a multipart upload. The minimum value is `5242880` (5 MiB). The default value is `10485760` (10 MiB). | | `bulkLoadS3MultipartThreshold` | The size, in bytes, above which the driver uses multipart upload instead of a single PUT request. The default value is `10485760` (10 MiB). | | `bulkLoadS3Prefix` | The key prefix for staged objects in S3. The default value is `ocient-bulk-load/`. | | `bulkLoadS3Region` | The AWS region for the S3 endpoint. The default value is `us-east-1`. | | `bulkLoadS3SecretAccessKey` | The secret access key for authenticating to the S3 endpoint. This parameter is required when you set the `bulkLoadMode` parameter to `s3`. | | `bulkLoadS3UploadConcurrency` | The number of data chunks to upload to S3 in parallel. Increasing this value can improve staging throughput for large batches. The default value is `1`. | | `bulkLoadSshChannelTimeoutSeconds` | The number of seconds to wait for an SFTP channel to open on an established SSH connection during bulk load. If the channel does not open within this duration, the driver throws an error. The default value is `30`. | | `bulkLoadSshConnectTimeoutSeconds` | The number of seconds to wait for the SSH connection and authentication to a Loader Node during bulk load. If the system does not establish a connection within this duration, the driver throws an error. The default value is `10`. | | `bulkLoadSshKeyPath` | The absolute path to the password-less private SSH key file. The default value is `~/.ssh/id_rsa`. | | `bulkLoadSshHostKeyVerification` | Controls how the JDBC driver verifies an SSH server host key for bulk load connections.

Supported values are:

`acceptAll` (default) — The driver accepts all host keys. The connector does not check the server host key against the `known_hosts` file and does not output warning logs.
`strict` — The driver accepts only hosts with keys present in the `known_hosts` file (see the `bulkLoadSshKnownHostsPath` parameter). The driver rejects unknown hosts or hosts with changed keys.
`acceptNew` — The driver accepts new hosts and adds their keys to the `known_hosts` file, but rejects hosts with keys changed from the data in `known_hosts`. | | `bulkLoadSshKnownHostsPath` | The path to the `known_hosts` file, which contains SSH verification keys. The system uses this path only if the `bulkLoadSshHostKeyVerification` parameter is set to `strict` or `acceptNew`.

The default path is `~/.ssh/known_hosts`. | | `bulkLoadSshUser` | The SSH username to use when connecting to Loader Nodes. | | `bulkLoadThreshold` | The minimum number of rows in a batch group to trigger a bulk load. The default value is `25000`. | | `bulkLoadUseAtomicPipeline` | If you set this parameter to `true`, bulk load uses an atomic pipeline that creates, starts, and monitors the pipeline in a single blocking operation `(CREATE TRANSACTIONAL PIPELINE ... START FOREGROUND)`. This mode requires the Ocient System version 27.1 or later.

Set this parameter to `false` to use the legacy create, start, monitor, and drop functionality when connecting to older servers. The default value is `true`. | | `defaultSchema` | Default schema. | | `enableBulkLoad` | Enables the high-speed bulk load feature. Set this parameter value to `true` to enable this feature. Otherwise, set this parameter to `false` to leave the feature disabled. The default value is `false`. | | `force` | If set to true (case-sensitive), this parameter disables load-balancing for the connection. | | `handshake` | Specifies the handshake protocol used for the connection.
Supported options include: `"CBC", "GCM", "SSO"`
`"GCM"` — (Galois/Counter Mode). This is the default encryption and is the recommended password encryption algorithm.
`"CBC"` — (Cipher Block Chaining) for password encryption.
`"SSO"` — Single Sign-On. | | `identityprovider` | An SSO integration established in the database. For details, see [CREATE SSO INTEGRATION](/cluster-and-node-management#create-sso-integration). | | `logfile` | The filename to use for JDBC tracing. | | `loglevel` | If set to ERROR (warnings and errors only) or DEBUG (verbose tracing) and logfile is also set, JDBC tracing is enabled. This parameter is case-sensitive. | | `longQueryThreshold` | Estimated query runtime in milliseconds before deeper query optimization runs.
`0` — Use database server default.
`-1` — Never run deeper optimization. | | `maxRows` | Maximum allowed result set size in the number of rows. | | `maxRowsPerInsertBatch` | The maximum number of rows to combine into a single multi-row `INSERT` SQL statement when executing a batch load. If the total number of rows in a batch group exceeds this limit, the driver splits the group into multiple sub-statements.

The default value is 128,000 rows. | | `maxTempDisk` | Maximum allowed temp disk usage as a percentage (0 - 100). | | `maxThreadsPerResultSet` | The maximum number of threads the client uses to fetch rows from the server per Result Set, as defined in the [official JDBC documentation](https://docs.oracle.com/javase/7/docs/api/java/sql/ResultSet.html). In this context, a thread represents a logical connection to the database.
Applications that create and operate on Statement objects concurrently might find value in setting this parameter. Defaults to `0`, which effectively creates an unbounded Result Set thread pool. | | `maxTime` | Maximum allowed runtime of a query in seconds before it is canceled on the server. | | `networkTimeout` | Network connection timeout in milliseconds.
If unspecified, this defaults to 10000 milliseconds. | | `parallelism` | Limits a query to running on a specified number of cores on each CPU. | | `password` | The password for the user. | | `priority` | Query priority. This sets the priority for queries to run on the server side.
If unspecified, this defaults to `1.0`. | | `priorityAdjustFactor` | The default query priority adjustment value. For details, see [SET ADJUSTFACTOR](/commands-supported-by-the-ocient-jdbc-cli-program#set-adjustfactor). | | `priorityAdjustTime` | The default frequency to adjust the query priority. For details, see [SET ADJUSTTIME](/commands-supported-by-the-ocient-jdbc-cli-program#set-adjusttime). | | `serviceClassName` | Specifies the name of the service class to use for the database session. | | `sparkMode` | When you set this parameter to `true`, the driver enables -specific JDBC behavior intended to improve compatibility with the Ocient Spark connector. For details, see [JDBC Spark Connector](/jdbc-spark-connector).

This parameter defaults to `false` if you are connecting directly using the Ocient JDBC driver (e.g., connecting with the `DriverManager` class or CLI).
Otherwise, if you are using the Spark connector (catalog or `.format("ocient")`), the default is `true`. | | `ssoNumericAddress` | Specifies the SSO callback URL as `127.0.0.1`. When this value is `false`, the URL is `localhost`. The default value is `false`. | | `ssoOAuthCodeCallbackPort` | If `ssoOAuthFlow=authorizationCode`, this parameter specifies the port that the SSO authorization uses.
If unspecified, the default value is `7050`. | | `ssoOAuthFlow` | This parameter is only applicable if SSO authorization is enabled.
Forces the driver to use either the "authorizationCode" or "deviceGrant" flow to establish a Single Sign-On session.
If this parameter is not provided, the System uses the "authorizationCode" flow when a web browser is available to the client and the "deviceGrant" flow when a web browser is not available to the client. | | `ssoDebugMode` | When set to `true`, the system records additional log messages related to SSO. | | `ssoOktaNativeTokenPath` | The path to the Native Token (AES-256GCM JWE). The path is relative to your home directory (i.e. '\~/'). | | `ssoSslCallback` | Specifies whether to use HTTPS instead of HTTP for the SSO callback URL. The default value is `false`. | | `ssoTimeoutSeconds` | The number of seconds before the SSO connection request times out. The default value is `60`. | | `statementPooling` | ℹ️ This parameter has been removed as of Ocient JDBC version 2.104 and later.
When set to `ON`, recently used statements are cached.
Set to `OFF` when using a third-party application that implements statement pooling | | `tls` | Can be set to `unverified` or `on`. Enables SSL/TLS encryption for the connection. | | `timeoutMillis` | Number of milliseconds before cancellable operations are timed out and killed by the driver. 0 means no timeout. Default: 0 | | `user` | The identifier of the user. | ### JVM System Properties The Ocient JDBC driver supports system properties that control driver-wide behavior. Unlike [connection properties](#supported-jdbc-connection-properties), JVM system properties apply globally to all connections within the JVM. To set JVM properties, use the `-D` flag from the `java` command line. The system reads these properties once when the driver initializes and cannot change them at runtime. **Example** This example launches the JDBC CLI with the memory throttle disabled. There is no space between `-D` and the property name. ```shell Shell theme={null} java -Dcom.ocient.jdbc.rs.disable-memory-throttle=true \ -classpath ocient-jdbc4-jar-with-dependencies.jar com.ocient.cli.CLI ``` #### Supported JVM System Properties | **Property** | **Type** | **Default** | **Description** | | -------------------------------------------- | -------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `com.ocient.jdbc.rs.disable-memory-throttle` | Boolean | `false` | Controls the heap-based result-set fetch throttle.

When you set this property to `false` (default), the driver pauses new `FETCH_DATA` requests when the projected heap usage exceeds 90% of `Runtime.maxMemory()`, waiting for memory to free before resuming. This action prevents the `OutOfMemoryError` error in most cases.

When you set it to `true`, the driver does not throttle on heap usage. If memory is genuinely exhausted, the JVM throws the `OutOfMemoryError` error instead of the driver waiting. | JVM system properties are not part of the JDBC connection URL or the `Properties` object the system passes to `DriverManager.getConnection()`. You must set them at the `java` command line. Set the `com.ocient.jdbc.rs.disable-memory-throttle` property to `true` only when you have verified that your application has sufficient heap headroom or when the throttle interferes with the expected workload behavior. Disabling the throttle can cause the `OutOfMemoryError` error if the JVM heap is exhausted. ### JDBC Bulk Loading For very large batches, the Ocient JDBC driver provides a high-speed bulk load path. When you enable bulk loading, the driver bypasses the standard multi-row insert operation and instead stages the batch data in the [JSON Lines](https://jsonlines.org/) format and loads it through a temporary, system-generated Ocient pipeline. The driver supports two transport modes for staging data. * **SSH/SFTP (default)** — The driver opens an SSH connection to a Loader Node and writes JSON data files directly to the node file system using SFTP. * **S3** — The driver uploads JSON data files to an S3-compatible bucket. The Ocient System then reads the staged files from S3 through standard pipeline mechanics. This mode eliminates the requirement for SSH access between the client and the Loader Nodes. Set the `bulkLoadMode` connection parameter to select the transport mode. The default value is `ssh`. #### Configuration Follow these steps to configure bulk loading in your JDBC driver. **Enable Bulk Loading** The JDBC driver disables bulk loading by default. To enable it, you must meet these conditions. * Set the `enableBulkLoad` connection parameter to `true`. * Use a parameterized `INSERT` statement that uses placeholders for values, e.g., `INSERT INTO customers (id, name, status) VALUES (?, ?, ?)`. For details, see [Set Up Parameterized Insert Statements](#set-up-parameterized-insert-statements). * The total number of rows in the batch group meets or exceeds the `bulkLoadThreshold` parameter. For recommended configuration settings for workloads, see [Bulk Loading Best Practices](/jdbc-spark-connector#bulk-loading-best-practices). **Choose a Transport Mode** Set the `bulkLoadMode` connection parameter to select how the driver stages data for the pipeline. The supported values are `ssh` (default) and `s3`. **SSH Mode Configuration** SSH mode requires passwordless SSH access from the client to all Loader Nodes in the cluster. To point the driver to your private SSH key file, set the `bulkLoadSshKeyPath` connection parameter. The client application must be able to read this file. The driver automatically discovers available Loader Nodes by querying the [sys.nodes](/system-catalog#sys-nodes) and [sys.service\_roles](/system-catalog#sys-service_roles) system catalog tables. **S3 Mode Configuration** The S3 mode stages data in an S3-compatible object store (such as S3 or ) and does not require SSH access to the Loader Nodes. When you select the S3 mode, you must provide the following required connection parameters. | **Parameter** | **Description** | | --------------------------- | ---------------------------------------------------------------------------- | | `bulkLoadS3Endpoint` | The S3-compatible endpoint URL (e.g., `https://s3.us-east-1.amazonaws.com`). | | `bulkLoadS3Bucket` | The S3 bucket name for staging data files. | | `bulkLoadS3AccessKeyId` | The access key identifier for authenticating to the S3 endpoint. | | `bulkLoadS3SecretAccessKey` | The secret access key for authenticating to the S3 endpoint. | You can also set optional S3 parameters to control the region, key prefix, path-style access, multipart upload behavior, upload concurrency, and API call timeout. For the full list of S3 parameters and their defaults, see the [Supported JDBC Connection Properties](#supported-jdbc-connection-properties) table. The driver uploads JSON data chunks to the staging bucket, generates a `CREATE TRANSACTIONAL PIPELINE SOURCE S3` SQL statement that points the Ocient System to the staged objects, and monitors the pipeline to completion. After the pipeline finishes, the driver deletes the staged objects from S3 and drops the pipeline. On failure, this cleanup occurs only when you set the `bulkLoadCleanupOnFailure` parameter to `true` (the default). The S3 access key identifier and secret access key are embedded in the pipeline DDL so that the Ocient System can read the staged objects. The pipeline is transactional and the driver drops it immediately after use. The Spark connector redacts credentials from its logs. **Data Type Mapping** The JDBC driver supports all standard scalar types. For complex types, Java SQL STRUCT (`java.sql.Struct`) types load as Ocient TUPLE types, and Java ARRAY (`java.sql.Array`) types load as Ocient ARRAY types. #### Set Up Parameterized `INSERT` Statements These steps demonstrate how to use a single parameterized `INSERT` statement using the Java `PreparedStatement` class. The parameterized statement binds different values for each row. Create the `PreparedStatement` object `ps` with `?` placeholders. ```java Java theme={null} String sql = "INSERT INTO customers (id, name, status) VALUES (?, ?, ?)"; PreparedStatement ps = conn.prepareStatement(sql); ``` Bind parameter values by position. ```java Java theme={null} ps.setLong(1, 123L); // First ? ps.setString(2, "Alice"); // Second ? ps.setString(3, "ACTIVE"); // Third ? ``` Execute a single insert. ```java Java theme={null} ps.executeUpdate(); ``` Or, add multiple rows as a batch. ```java Java theme={null} // First row ps.setLong(1, 123L); ps.setString(2, "Alice"); ps.setString(3, "ACTIVE"); ps.addBatch(); // Second row ps.setLong(1, 124L); ps.setString(2, "Bob"); ps.setString(3, "INACTIVE"); ps.addBatch(); // Send them together int[] results = ps.executeBatch(); ``` Close the resources. ```java Java theme={null} ps.close(); ``` ### Connection Encryption (SSL/TLS) The JDBC driver can use SSL/TLS to connect to Ocient, causing all traffic to be encrypted. Specify the `tls` property on the connect statement to enable TLS support. The `tls` property supports these values. **unverified** Traffic on the connection is encrypted, but no verification is done on the certificate received from the Ocient System. **on** Traffic is encrypted, and the JDBC client must be able to verify the certificate received from the Ocient System. The TLS `on` mode requires that the client knows the **Certificate Authority** that signed the certificate provided by the Ocient System. Typically, this mode requires either that the certificate is signed by a well-known certificate authority, or the Certificate Authority certificate has been imported into the truststore of the Java system. The Java `keytool` utility is used to manipulate a Java truststore. [Secure Connections Using TLS](/secure-connections-using-tls) discusses how you can configure user-defined certificates for the Ocient System. ### Sample Java Program Using the Ocient JDBC Driver This sample program demonstrates how to utilize the Ocient JDBC driver to establish a connection to a database, construct a prepared SQL statement, execute the query, and iterate through the result set. ```java Java theme={null} public class OcientJDBCExample { public static void main(final String args[]) { Class.forName("com.ocient.jdbc.JDBCDriver"); Properties props = new Properties(); props.setProperty("user", "username"); props.setProperty("password", "pwd"); props.setProperty("force", "true"); String url = "jdbc:ocient://192.168.121.82:4050/db"; Connection conn = DriverManager.getConnection(url, props); PreparedStatement pstmt = conn.prepareStatement( "select l_orderkey from tpch.lineitem where l_linenumber = ?"); Pstmt.setInt(1, 4); ResultSet rs = pstmt.executeQuery(); while(rs.next()){ // do something with row } rs.close(); pstmt.close(); conn.close(); return; } } ``` For supported classes and methods, see [JDBC Classes and Methods](/jdbc-classes-and-methods). ### Run a Transaction Using the Ocient JDBC Driver To run a multi-statement transaction with the Ocient JDBC driver in a Java program, disable the autocommit mode using `setAutoCommit(false)`, and then call the `commit()` or `rollback()` methods. For transactions, autocommit mode is on by default. Ensure that you execute SQL statements that are supported by transactions. Otherwise, the database throws an error. For a list of supported statements, see [Transactions](/transactions). This example code connects to a sample database and creates the `public.txn_demo` table. Then, the code inserts two rows and displays the row count in the table. The code commits the INSERT statements and re-runs the row count. This code inserts two more rows and displays a row count. Then, the example rolls back the transaction and displays a row count. Finally, the code enables the autocommit mode. ```java Java theme={null} import java.sql.*; import java.util.Properties; public class TxnApi { static void show(Connection c, String label) throws SQLException { try (Statement s = c.createStatement(); ResultSet rs = s.executeQuery("SELECT count(*) AS n FROM public.txn_demo")) { rs.next(); System.out.println(label + " -> row count = " + rs.getInt("n")); } } public static void main(String[] args) throws Exception { String url = "jdbc:ocient://db.example.com:4050/salesdb"; Properties p = new Properties(); p.setProperty("user", "admin@system"); p.setProperty("password", "admin"); p.setProperty("tls", "unverified"); try (Connection c = DriverManager.getConnection(url, p)) { // DDL must run outside a transaction (autocommit on) try (Statement s = c.createStatement()) { s.execute("DROP TABLE IF EXISTS public.txn_demo"); s.execute("CREATE TABLE public.txn_demo (id INT, v VARCHAR(64))"); } // Transaction 1: multiple inserts and read-your-writes, then commit c.setAutoCommit(false); try (Statement s = c.createStatement()) { s.executeUpdate("INSERT INTO public.txn_demo VALUES (1, 'alpha')"); s.executeUpdate("INSERT INTO public.txn_demo VALUES (2, 'beta')"); show(c, "T1 in-txn (read-your-writes, expect 2)"); } c.commit(); show(c, "T1 after COMMIT (expect 2)"); // Transaction 2: inserts, then rollback try (Statement s = c.createStatement()) { s.executeUpdate("INSERT INTO public.txn_demo VALUES (3, 'gamma')"); s.executeUpdate("INSERT INTO public.txn_demo VALUES (4, 'delta')"); show(c, "T2 in-txn (expect 4)"); } c.rollback(); show(c, "T2 after ROLLBACK (expect 2)"); c.setAutoCommit(true); } } } ``` ## Related Links [Connect Using JDBC](/connect-using-jdbc) [Data Extract Tool](/data-extract-tool) [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program) [JDBC Classes and Methods](/jdbc-classes-and-methods). *** *Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.* # JDBC Release Notes Source: https://docs.ocient.com/jdbc-release-notes Release notes for the Ocient JDBC driver, including new features, bug fixes, compatibility changes, and upgrade guidance for each driver version. All JDBC drivers are located the [Ocient Maven repository](https://mvnrepository.com/artifact/com.ocient/ocient-jdbc4). ## 4.2.1 (2026-06-26) Internal updates only. ## 4.2.0 (2026-06-16) * Added transactional support. You can now group multiple statements into a single transaction by disabling autocommit and ending the transaction with the `COMMIT` or `ROLLBACK` command. For details, see [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). ## 4.1.0 (2026-04-01) * Updated the input parameter for the `CANCEL` and `KILL` commands to be a quoted string for the Universally Unique IDentifier (UUID), such as `'c87b506f-66d6-4987-8bf7-1fd4a06b64b1'`. * Added these new connection properties: `bulkLoadSshHostKeyVerification`, `bulkLoadSshKnownHostsPath`, and `maxRowsPerInsertBatch`. For details, see the [JDBC Connection Properties](/jdbc-manual#supported-jdbc-connection-properties). * You can now configure JDBC CLI configuration options for the `PERFORMANCE` and `TIMING` commands. For details, see the [JDBC Manual](/jdbc-manual). * Added these new extract options to the Data Extract Tool. For details, see [Data Extract Tool](/data-extract-tool): * FILE\_PREFIX\_EXISTS * FILE\_TYPE * NUM\_FETCH\_QUERIES * PARTITION\_MODE * PARTITION\_COLUMNS * SUCCESS\_MARKER * TARGET\_FILE\_SIZE\_MB ## 4.0.0 (2026-02-05) * Added the Connector that enables the Ocient System for Spark workloads. The connector allows Spark to read from and write to Ocient tables using Spark APIs and SQL statements. For details, see [JDBC Spark Connector](/jdbc-spark-connector). * Enabled SSO-authenticated users to access multiple databases using a single identity provider configuration. * Added graph functionality using the `OCGraph` class. For details, see [OCGraph Java Library](/ocgraph-java-library). ## 3.7.0 (2025-11-21) * Added quotes around `VARCHAR` data types in tuples and changed delimiter characters to parentheses. Now, the tuple output is `("foo", 5)` instead of `<>`. ## 3.6.7 (2025-10-11) * CLI is no longer replacing or removing any whitespace characters inside quoted fields. ## 3.6.6 (2025-10-01) * Enabled the `source` command to handle files containing the `connect to` command. ## 3.6.5 (2025-09-25) Internal updates only. ## 3.6.4 (2025-09-10) * Added the `printUuid` setting to the `.ocient-cli-configuration` file that determines whether the CLI automatically prints the Universally Unique IDentifier (UUID) of the query after the submission of the query by the user. The values are `on` and `off`. If you do not specify this setting, the default value is `off`. * Re-enabled asynchronous updates. Now, the system executes updates asynchronously by default. ## 3.5.1 (2025-06-16) * Disabled asynchronous updates. ## 3.5.0 (2025-06-07) * Added support for asynchronous updates, enabling you to retrieve the query identifier immediately after executing the SQL statement instead of when it finishes on the server. Now, you can use Ctrl-C to cancel the update at the command line. * Added support for graph algorithms using vertex and edge table representations. These algorithms cover the same capabilities as GraphX from . * Added the `DISTRIBUTE_WORK` connection parameter, which distributes queries or commands between all endpoints in a round-robin way, instead of relying on server-side load balancing. ## 3.4.1 (2025-05-14) * New Features * Changed the default value for the `path-style-access` extract option to `true`. ## 3.4.0 (2025-04-30) * New Features * Removed quotes added during the extract around `VARCHAR` data types inside arrays by default. Added the `QUOTE_ARRAY_VARCHARS` option for the extract that, when you set the value to `true`, adds those quotes back. * Removed the leading forward slash from IP addresses extracted from the database. * Added periodic polling of the `.ocient-cli-configuration` file, so the system applies changes to the settings in the file at runtime. The default interval is 5 minutes. You can change the default value by setting the `com.ocient.cli.config-poll-interval` system property to the chosen number of seconds. * Fixed the CLI result set output from formatted to a raw string. ## 3.3.1 (2025-03-30) Internal updates only. ## 3.3.0 (2025-03-06) * New Features * Added the specification of the identity provider in the connection string when connecting to the database. * Added support for the bzip2 and xz compression. * Added support for multi-line definitions in SQL statements. * Changed the default SSO callback address to `http://localhost:7050/ocient/oauth2/v1/callback`. * Added the `ssoNumericAddress` parameter to force the client to use the IP address `127.0.0.1` instead of `localhost` when you set this parameter to `true`. * Enabled switching the metrics target directory without having to restart the CLI session. * Added support for line and block comments in multi-line SQL statements. ## 3.2.4 (2025-01-03) Internal updates only. ## 3.2.3 (2024-12-13) Internal updates only. ## 3.2.2 (2024-11-08) Internal updates only. ## 3.2.1 (2024-09-30) * New Features * Changed the default escape character for quotes in VARCHAR arrays from `\` to `"` for extracts. * Added connection property to redact usernames and SQL text in JDBC logs. * Increased security for passwords in the command-line interface and history file. ## 3.1.7 (2024-06-03) Internal updates only. ## 3.1.6 (2024-06-03) Internal updates only. ## 3.1.5 (2024-06-03) **Bugs** * \[DB-28722]: Fixed the issue where the client closed the result set before sending the KILL\_QUERY request. * \[DB-28325]: Fixed an underlying SLF4JLogger error that prevented the driver from importing. * \[DB-24984]: Fixed HTTP client configuration to accept standard cookie specifications. ## 3.1.1 (2024-03-08) * *New Features* * Enabled support of fetching cached queries using the JDBC 3.1.1 driver with version 22.1 and older versions of the . * Fixed connection failures by adding sequence numbers to metadata responses on the client side for legacy databases without sequence number support. ## 3.1.0 (2024-01-31) * *New Features* * Improved maximum throughput between client and server, specifically for transferring large and sorted result sets. * Deprecated `com.ocient.jdbc.XGConnection#redirect`. The redirect method in the `com.ocient.jdbc.XGConnection` class is now deprecated and is slated for removal in future releases. For applications that require a connection to a specific SQL Node, Ocient recommends to use the `force=true` connection property. ## 3.0.0 (2023-10-24) * *New Features* * Improved scalability for high-volume concurrent database connections for applications such as and Superset. * Improved latency for bulk data transfers. ## 2.106 (2023-06-29) * *New Features* * Fixed requirement to restart the application after cluster upgrade. ## 2.105 (2023-03-22) * *New Features* * Improved performance by pooling database connections used for sideband channels. * Added `MAX_THREADS_PER_RESULTSET` connection property. ## 2.100 (2023-03-10) * *New Features* * Added ability to periodically log metrics to a file. * Added `METRICS ON ` CLI command. ## 2.99 (2023-01-24) * *New Features* * Added `PERFORMANCE NETWORK` CLI command. * Added `PERFORMANCE DATABASE` CLI command. * Deprecate `PERFORMANCE ON` CLI command. ## 2.98 (2023-01-17) * New Features * Removed OpenJump system dependency ## 2.97 (2022-12-27) * New Features * Added support for `DELETE FROM` ## 2.96 (2022-12-21) * New Features * Fixed `IndexOutOfBoundsException` in server redirect ## 2.95 (2022-12-07) * New Features * Log stack trace of `XGStatement#sendAndReceive()` read errors ## 2.94 (2022-12-07) * New Features * Allow extract to `/dev/null` ## 2.93 (2022-11-28) * New Features * Strip trailing statement delimiter from queries ## 2.92 (2022-11-22) * New Features * Include `jdbc.sql.Driver service in META-INF` ## 2.91 (2022-11-22) * New Features * Add support for `REFRESH MLMODEL` command, which allows updating of a Machine Learning model without changing the ML model properties ## 2.90 (2022-11-22) * \[DB-22501] Never resubmit queries that have been sent to the server ## 2.89 (2022-11-18) * Add support for EXPORT MLMODEL command ## 2.88 (2022-11-08) * Load JDBC version from build properties file at runtime ## 2.87 (2022-11-07) * Set Service Class ## 2.86 (2022-10-24) * Allow connection property override ## 2.85 (2022-10-24) * Address CVE-2022-33980 and CVE-2022-24823 ## 2.84 (2022-10-24) * Print hex instead of address of byte array in XGTuple#toString ## 2.83 (2022-10-18) * Re-enabled ability to override a connection property * Fix query timeouts which were not being honored * Handle non-resolvable database DNS names received from the database ## 2.82 (2022-10-14) * Made JDBC Connection properties case-insensitive using a new class CaselessProperties * Converted the rest of the connection properties to the ConnectionProperty enum ## 2.81 (2022-10-03) * Implement XGPreparedStatement#setObject(Index, Timestamp, Calendar) * Add "statementPooling" driver property. Set to "OFF" to disable statement caching ## 2.80 (2022-10-03) * Add `GET JDBC COMMIT INFO` command * Add `/version`, `/commit`, `/info`, and `/pid` metrics endpoints ## 2.79 (2022-10-03) * Ensure backwards compatibility with JRE8 ## 2.78 (2022-09-19) * Add CLI support for `EXPORT PIPELINE` ## 2.77 (2022-09-19) * Add `START PIPELINE` and `STOP PIPELINE` for updates ## 2.76 (2022-09-02) * Made `SHOW` command work as a SELECT outside of just the CLI ## 2.75 (2022-09-02) * Made `SHOW` command a select command rather than an update command ## 2.74 (2022-08-29) * `BENCHMARK` command consumes result set for consistency w/ manual input ## 2.73 (2022-08-26) * New Features * Add support for `SHOW` command, which allows easy querying of the information schema. ## 2.72 (2022-08-23) * New Features * \[DB-20912] Add support for one-time password (OTP) based SSO flows * \[DB-20912] Add OAuth2 Authorization Code w/ PKCE Single Sign-On Flow * \[DB-20912] Add OAuth2 Device Authorization Grant Single Sign-On Flow for input constrained environments ## 2.71 (2022-08-23) * New Features * Add `BENCHMARK NSAMPLES COMMAND` which will run `COMMAND` `NSAMPLES` times and produce a timing report in the CLI output. * Add `PRINTUUID [ON | OFF]` which will print the query UUID to the CLI output. ## 2.70 (2022-08-11) * New Features * \[DB-21078] Add `s3_upload_part_size` and `s3_upload_part_parallelism` extract configuration properties. ## 2.69 (2022-08-11) * New Features * \[DB-21078] Add REST endpoint to expose query and data extract performance metrics * \[DB-21078] Add ms precision to JDBC log entries ## 2.68 (2022-08-09) * New Features * \[DB-21251] Allow extract property override via "JDBC\_EXTRACT\_\*" prefixed environment variable ## 2.67 (2022-06-30) * New Features * Allow aliases on nested queries in extract SQL statements (DB-21075) ## 2.66 (2022-06-27) * New Features * Escape escape characters in quoted field for non-comma delimited extracts ## 2.65 * New Features * Add support for using `CANCEL TASK` to cancel a task instead of treating it like a `CANCEL` command ## 2.64 (2022-06-21) * New Features * Resolve CVE-2022-24823; update netty-common version (4.1.77.Final) * Resolve CVE-2021-43797; update netty-codec-http version (4.1.71.Final) ## 2.63 (2022-06-01) * New Features * Replace `skip_header` with `header_mode` in CSV extract options. * Add CLI configuration file, `~/.ocient-cli-configuration`, and idle timeout option `cliIdleTimeoutMinutes` ## 2.62 (2022-06-01) * New Features * Add `escape_unquoted_values`, `input_escaped`, and `quote_all_fields` in CSV extract options. * Add a new custom TSV writer that handles all non-comma delimiter cases. ## 2.61 (2022-05-31) * New Features * Add `compression_block_size`, `compression_level`, and `num_compression_threads` Data extract properties. ## 2.60 (2022-05-24) * New Features * Allow non-query commands to print results when performance mode is turned on for easier debugging. ## 2.59 (2022-05-17) * New Features * Add UTF-8 character translation to Data extract tool. ## 2.58 (2022-4-20) * New Features * Prevent shell expansion of `!` when reading user input from CLI ## 2.57 (2022-3-17) * New Features * Bumped protocol version for quiesce * Modified length checks to check if higher order bit is set, signalling a message from the server ## 2.56 (2022-3-15) * New Features * Added SQL states for EXECUTE\_PLAN\_AUTH\_FAILURE and EXECUTE\_INLINE\_PLAN\_AUTH\_FAILURE ## 2.55 (2022-3-11) * New Features * Reconnect and rerun if writing into socket fails when sending request * Reconnect and rerun if when sending request, shows -1 length * Added checks for -1 length on response for server signalling a quiescing connection close * Copy secondary interfaces and secondary index when returning a cached connection ## 2.54 (2022-3-9) * New Features * Added CLI support for SET ADJUSTFACTOR, ADJUSTTIME * Added new columns in system queries: sys.queries: initial\_priority, initial\_effective\_priority, effective\_priority, priority\_adjust\_factor, priority\_adjust\_time sys.completed\_queries: initial\_priority, initial\_effective\_priority, final\_effective\_priority, priority\_adjust\_factor, priority\_adjust\_time ## 2.53 (2022-2-8) * New Features * Added redirect support for execute export, explain pipeline, check data ## 2.52 (2022-1-18) * New Features * Secondary result set threads will start fetching immediately after the first result set thread gets a ping. ## 2.51 (2022-1-14) * New Features * Make force external consistent with other connection parameters ## 2.50 (2022-1-13) * New Features * Fix invalid argument message to not refer to ODBC client ## 2.49 (2022-1-6) * New Features * Add more logging to multithreaded result set fetch. ## 2.48 (2022-1-3) * New Features * Insert statements supported ## 2.47 (2021-12-16) * New Features * Data extract tool released. * Deactive connection caching for calls to createConnection ## 2.46 (2021-12-8) * New Features * Fix connection pooling handling with regards to setting schema incorrectly. ## 2.45 (2021-12-2) * New Features * Swallow runtime error as well when trying to set socket options. ## 2.44 (2021-12-1) * New Features * Wrap invalid socket options in try catch block and swallow exceptions. For compatability with Java versions that are older. ## 2.43 (2021-11-22) * New Features * Change driver to not rely on the pom/MANIFEST for the driver version. Move the version into JDBCDriver ## 2.42 (2021-11-18) * New Features * Changed connections to utilize keep alive probes in order to detect dead connections. ## 2.41 (2021-11-07) * New Features * Upgrade Jline version to 3.21 ## 2.40 (2021-11-03) * New Features * Improve result set fetching performance ## 2.39 (2021-10-29) * New Features * Fix handling of expired token being used to start a session ## 2.38 (2021-10-27) * New Features * Updated auth related SQLState error codes * Fix handling of result set close during result set caching ## 2.37 (2021-10-25) * New Features * Force redirect support for testing purposes. * Removed duplicate code from CLI ## 2.36 (2021-10-14) * New Features * Implemented refresh sessions support. * Implement get server session ID. ## 2.35 (2021-10-11) * New Features * ExecutePlan and ExecuteInlinePlan bugfixes ## 2.34 (2021-9-9) * New Features * Add token signature and issuer fingerprint fields to sso token handshake. * Add explicit SSO flags to handshake GCM. * Pass networkTimeout to new connection upon copy. ## 2.33 (2021-9-7) * New Features * Add support for explicit SSO handshakes * Revert executeQuery redirection change ## 2.32 (2021-9-1) * New Features * CLI source command stops on error * Support "output next query append" * Add statement command "get jdbc version" ## 2.31 (2021-8-30) * New Features * Add sso handshake support * Refactor duplicated code in handshakes for saving secondary interfaces ## 2.30 (2021-8-26) * New Features * Add validation on the driver side for set parameter settings. * Pass up invalid set parameter command errors. ## 2.29 (2021-8-20) * New Features * Fix the way connection resets parameters when reset is true. * Fix an issue where redirect was not properly using mapped secondary interfaces ## 2.28 (2021-8-16) * New Features * Fix sending parameters on reset. maxrow, maxtempdisk, maxtime, parallelism, priority now gets reset properly. * Map list all queries to select \* from sys.queries * Improve logging * Statements with closed connections will not get returned to the cache ## 2.27 (2021-8-10) * New Features * Fixed poorly structured if statement in CLI source command. * Remove more stack trace printing and add more logs. ## 2.26 (2021-7-30) * New Features * Move the duplicated regex code in CLI.java and XGStatement.java into a new file. * Implement a generic regex for syntax checking the set family of sql commands. * Fix a bug in resetting commands using lower cases. "set maxrow reset;" does not work. Needs to be capitalized. ## 2.25 (2021-7-27) * New Features * Default clientVersion to 0.00 if the driver provides a null clientVersion to XGConnection. ## 2.24 (2021-7-12) * New Features * Have setSchema handle unquoted caps sql. "SET SCHEMA MADISON" will set schema to "madison". ## 2.23 (2021-7-6) * New Features * Change SET CONCURRENCY command to SET PARALLELISM * Change "concurrency" driver property to "parallelism" ## 2.22 (2021-7-1) * New Features * Add openJump extensions to jdbc jar. * Fix source command for plan execute inline. ## 2.21 (2021-6-30) * New Features * Fix new hashcode for null elements. ## 2.20 (2021-6-28) * New Features * Update list all queries metadata to match that of select \* from sys.queries * Add more connection level settings to hashCode. ## 2.19 (2021-6-26) * New Features * Improve source command capabilities and performance ## 2.18 (2021-6-26) * New Features * Save command history across sessions ## 2.17 (2021-6-25) * New Features * Include the currently set schema to the connection hashcode. ## 2.16 (2021-6-23) * New Features * Fix another issue with timeoutMillis. The timeout will now work correctly with zero ping buffers. ## 2.15 (2021-6-21) * New Features * For dBeaver compatability, not setting a user in driver properties will cause the driver to default to empty string. ## 2.14 (2021-6-17) * New Features * Fix timeoutMillis by correctly inheriting timeoutMillis from properties. * Clear warnings before running executeQuery and executeUpdate. ## 2.13 (2021-6-09) * New Features * Added command to SET PSO SEED for the random number generator used in PSO * Remove chatty number of rows log. * Add remoteIp and service class to list all queries. * Add sleep\_in\_optimizer command for testing. ## 2.12 (2021-5-26) * New Features * Fix the parsing for execute() to properly route to executeStatement() ## 2.11 (2021-5-20) * New Features * Improve cache and statement pooling support for redirection. * Fix timestamp and time with negative nanos. ## 2.10 (2021-5-19) * New Features * Added 10 different colors for GIS types in KML * Added non-GIS types to description of each GIS object in KML * Moved KML feature from CLI to driver ## 2.09 (2021-5-18) * New Features * Use daemon threads for background caching tasks. This allows the program to exit without finishing these tasks. ## 2.08 (2021-5-10) * New Features * Add versions packaging to pom.xml. * Updated dependencies to latest version using versions plugin. * Change TLS to unverified by default. ## 2.07 (2021-5-05) * New Features * Add OBJECT\_NOT\_FOUND\_WARN warning. * Add OBJECT\_ALREADY\_EXISTS\_WARN warning. * Fix some misleading log messages. ## 2.06 (2021-4-29) * New Features * Add a log for if reconnect() fails to close a socket. * Driver sends over unique session ID to server. * Fix a misleading log in client handshake. ## 2.05 (2021-4-28) * New Features * Fix spotbugs and narrow spotbugs filter. ## 2.04 (2021-4-22) * New Features * Added getters to GIS types ## 2.03 (2021-4-22) * New Features * Output next query also prints all gis types into a kml file ## 2.01 (2021-4-21) * New Features * Fix empty point for STPoint class ## 2.01 (2021-4-20) * New Features * Better build integration with xgsrc * Upgraded protobuf to version 3.14 ## 2.00 (2021-4-17) * New Features * Handling for cache limit warning. ## 1.99 (2021-4-13) * New Features * Add support for EXPLAIN DEBUG, and make EXPLAIN format JSON by default ## 1.98 (2021-4-05) * New Features * Add support for QUARANTINE * Fix CLI command force external on ## 1.97 (2021-4-02) * New Features * Queries that exceed the row limit set by XGConnection::setMaxRows now silently omit excess rows. The previous behavior would result in query failure. ## 1.96 (2021-4-01) * New Features * Add support for POINT EMPTY * Fix CLI performance on/off affecting timing setting. ## 1.95 (2021-3-22) * New Features * Add support for CHECK DATA to CLI * Fix bug where defaultSchema was not working. ## 1.94 (2021-3-18) * New Features * Add major and minor version to client handshake. * Add logging to set param functions. ## 1.93 (2021-3-11) * New Features * Improve performance of writing query results to a file. ## 1.92 (2021-3-5) * New Features * Add driver method to cancel all cache return threads. * Add CLI functionality to limit max history size. ## 1.91 (2021-3-4) * New Features * Add support for st\_linestring and st\_polygon ## 1.90 (2021-3-2) * New Features * Enable spotbugs analysis * Increase tracing to millisecond granularity ## 1.89 (2021-2-24) * New Features * Switch to using AES/GCM/NoPadding encryption ## 1.88 (2021-2-24) * New Features * Fix some broken custom xgMetadata calls. ## 1.87 (2021-2-24) * New Features * Added support for TUPLE columns as a SQL Struct type ## 1.86 (2021-2-19) * New Features * Fix more OWASP bugs ## 1.85 (2021-2-18) * New Features * Added CLI support for EXPORT VIEW ## 1.84 (2021-2-16) * New Features * Update the request type for EXPLAIN PIPELINE ## 1.83 (2021-2-15) * New Features * Fixed a bunch of OWASP bugs. ## 1.82 (2021-2-12) * New Features * Add CLI support for SET MAXROWS, PRIORITY, CONCURRENCY, MAXTIME, and MAXTEMPDISK ## 1.81 (2021-2-12) * New Features * Fix a compiler error with CLUSTER\_NOT\_FOUND error ## 1.80 (2021-2-12) * New Features * Add CLUSTER\_NOT\_FOUND error ## 1.79 (2021-2-11) * New Features * Add spotbugs * Add support for EXPLAIN PIPELINE to CLI ## 1.78 (2021-2-5) * New Features * Add SqlStates for LUP decomposition * Add additional cache logging. * Fix incorrect schema generated when first statements are created. ## 1.77 (2021-2-5) * New Features * Remove unnecessary calls to fetchServerVersion * Fix schema for pooled connections and statement. ## 1.76 (2021-2-2) * New Features * Pooling improvements ## 1.75 (2021-2-1) * New Features * Allow changing session variable defaults via connection properties and make session variable overrides local to statement objects. ## 1.74 (2021-1-31) * New Features * Performance optimizations and restructuring ## 1.73 (2021-1-30) * New Features * The JDBC driver now does connection and statement pooling that is automatic and transparent to the caller. ## 1.72 (2021-1-27) * New Features * Add support for clearBatch() ## 1.71 (2021-1-14) * New Features * Add support for session overrides of service class limits * SET MAXROWS \{N} * SET MAXTIME \{N} * SET PRIORITY \{N} * SET CONCURRENCY \{N} ## 1.70 (2021-1-12) * New Features * Fix race condition between adding fetch threads to array and iterating. Also removed some duplicated code. ## 1.69 (2021-1-07) * New Features * Remove deprecated plan proto ## 1.68 (2020-12-24) * New Features * Fix describe view truncate in CLI. ## 1.67 (2020-12-14) * New Features * Fixed cols2Types mapping in RS returned by LIST ALL COMPLETED QUERIES. ## 1.66 (2020-12-10) * New Features * Add spaces to syntax parsing. ## 1.65 (2020-12-9) * New Features * Completed queries update. ## 1.64 (2020-12-7) * New Features * Properly handle the nullable property for the LIST ALL COMPLETED QUERIES command. ## 1.63 (2020-12-1) * New Features * Don’t print stack trace when select queries complete exceptionally. ## 1.62 (2020-11-30) * New Features * Added support for LIST ALL COMPLETED QUERIES command. ## 1.61 (2020-11-15) * New Features * Fix reconnect and resending logic. ## 1.60 (2020-11-10) * New Features * DESCRIBE TABLE now consistently uses SMALLINT and BIGINT to describe inner types of arrays ## 1.59 (2020-11-4) * New Features * remove another unnecessary driver version fetch. * add jar version to log ## 1.58 (2020-11-4) * New Features * fix fetch version redirect loop. ## 1.57 (2020-11-4) * New Features * make SQLException codes unique. ## 1.56 (2020-11-2) * New Features * kill and cancel now throws syntax errors. ## 1.55 (2020-10-28) * New Features * CLI now recognizes a DDL statement for INVALIDATE STATS. ## 1.54 (2020-10-21) * New Features * fix another spot where all the slq nodes can be brought down. ## 1.53 (2020-10-20) * New Features * exportTranslation will close the result set. ## 1.52 (2020-10-19) * New Features * executeQuery will not rerun query after reconnect. ## 1.51 (2020-10-11) * New Features * Add token fields and username to localQueries protobuf ## 1.50 (2020-10-11) * New Features * Multiple result set threads support is complete ## 1.49 (2020-10-06) * New Features * Experimental support for multiple result set threads ## 1.48 (2020-09-28) * New Features * Minor fix of SQLStates * setParms() does no work when there are no parameters ## 1.47 (2020-09-26) * New Features * Improve performance for large SQL statements ## 1.46 (2020-09-21) * New Features * Bug fix for driver not reconnecting when a previous result set is still open. ## 1.45 (2020-09-16) * New Features * Move export translation into result set. ## 1.44 (2020-09-15) * New Features * Support granting and revoking both privileges and role membership. ## 1.43 (2020-09-14) * New Features * Explain, listTables, listViews, getSchema, describeTable, describeView, explainPlan, listAllQueries, and exportTable. ## 1.42 (2020-09-2) * New Features * Move some custom functionalities into the driver including: get/set schema, list (system) tables, list views, describe table/views, list indexes, execute/explain plan, cancel/kill query, list all queries, export table, set max rows, set pso. ## 1.41 (2020-08-20) * New Features * Ability to set max output rows from the CLI ## 1.40 (2020-08-14) * New Features * SSL support. ## 1.37 (2020-07-26) * New Features * Timestamp and Time now have nanosecond precision ## 1.36 (2020-07-25) * New Features * JDBC driver can now handle hostnames sent back in interface list ## 1.35 (2020-07-23) * New Features * Capture initial connection IP and use as last resort for reconnect() ## 1.34 (2020-07-14) * New Features * Implement new time/date/timestamp subclasses to circumvent bugs in the standard versions ## 1.33 (2020-07-04) * New Features * Add support for load balancing with secondary SQL interfaces ## 1.32 (2020-06-26) * New Features * Add support for Export Translation to CLI ## 1.31 (2020-06-23) * New Features * Fix misbehavior with 'plan execute inline' using the CLI ## 1.30 (2020-06-22) * New Features * Deprecated the HDFS connection table, which was consolidated into the external connection table ## 1.29 (2020-06-18) * New Features * Creating an MLModel on no data is now an error, not a warning ## 1.28 (2020-06-11) * New Features * Allow one connection to be shared by multiple threads, as long as each thread uses its own Statement object ## 1.27 (2020-06-11) * New Features * Improve trace log formatting, including adding thread id ## 1.26 (2020-06-09) * New Features * Add support for weeks interval type ## 1.25 (2020-06-05) * New Features * Support for getTableTypes() ## 1.24 (2020-06-05) * New Features * Protobuf message changes for Add Column feature ## 1.23 (2020-06-02) * New Features * Add new error code: Value too large, which indicates that a column value is larger than internal limits ## 1.22 (2020-05-31) * New Features * Add calendar support * Bug Fixes * Properly handle case-insensitivity of column names on result sets ## 1.21 (2020-05-26) * New Features * Add tracing support ## 1.20 (2020-05-19) * Bug Fixes * DB-11119 - JDBC multi-host support was not implemented correctly ## 1.19 (2020-05-13) ## 1.18 (2020-05-11) ## 1.16 (2020-04-15) * Bug Fixes * DB-10687 - Fix for Kill/Cancel query ## 1.15 (2020-04-14) * Bug Fixes * DB-9928 - Fixes for SQL Array type ## 1.14 (2020-04-11) * New Features * Protocol support for the NULLS FIRST in the ORDER BY clause ## 1.13 (2020-03-20) * New Features * Binary support for the serialization of the following data types: ST\_POINT, UUID, IP and IPV4 * Implementation of the Array SQL type ## 1.12 (2020-03-01) * Bug Fixes * DB-10155 - Broadcast kill query and cancel query requests to all sql nodes. * New Features * DB-10137 - Implement JDBC cancel query. Now it is possible to kill query in dbeaver. * DB-10120 - JDBC driver support list of sql nodes to connect to * DB-10119 - Support for multiple IPs under the same DNS address # JDBC Spark Connector Source: https://docs.ocient.com/jdbc-spark-connector Integrate Apache Spark with Ocient using the JDBC Spark Connector to run scalable analytics on large data sets with high-performance read and write operations. The connector is a [Spark DataSourceV2](https://downloads.apache.org/spark/docs/2.3.1/api/java/index.html?org/apache/spark/sql/sources/v2/DataSourceV2.html) implementation that adapts an Ocient System to operate as a first-class source and sink for Spark workloads. Built on top of the Ocient JDBC driver, the connector allows Spark to read from and write to Ocient tables using Spark APIs and SQL statements. The connector implements Spark catalog and table interfaces so you can register Ocient as a catalog (for `CREATE TABLE`, `INSERT`, `SELECT`, and `SHOW TABLES` SQL statements) or use it for ad‑hoc reads and writes. ## Key Features The Ocient Spark connector includes these key features: * Read Pushdown — The connector accelerates reads by pushing column selection, filters (including on nested fields), aggregations, and queries that only need the first N rows down to the Ocient System while still letting Spark validate the final results. * Read Partitioning — The connector parallelizes reads by splitting data into multiple Spark partitions based on a partition column. For details, see [Read Partitioning Options](#read-partitioning-options). * [DataFrame](https://spark.apache.org/docs/latest/sql-programming-guide.html) Write Behavior and Save Modes — The connector controls how it writes DataFrames to Ocient tables by honoring Spark save modes to append (`Append`), truncate‑and‑replace (`Overwrite`), or fail on existing tables (`ErrorIfExists`). * Catalog Support — The connector exposes Ocient as a Spark catalog so you can use standard Spark SQL directly on an Ocient System. ## Prerequisites To use the Ocient Spark connector, your system must meet these software requirements. | **Software** | **Version** | | --------------------- | ------------------------------------------------ | | Ocient | Use Ocient System version 26.1 or later. | | Operating System (OS) | , , or .
Use the latest version of each OS. | | Apache Spark | Version 3.5 or later. | | Version 8 or later. | | | Ocient JDBC driver | Version 4.0 or later. | Additionally, you must have the `SELECT`, `INSERT`, `CREATE`, and `DELETE` user privileges for the specified database. For details, see [Data Control Language (DCL) Statement Reference](/data-control-language-dcl-statement-reference). ## Ocient Spark Connector Setup and Initial Use To start working with the Ocient Spark Connector, register the connector. Then, you can start executing SQL statements. ### Connector Registration For best results, first register the connector as a catalog in Spark. To register the connector, edit the `spark-defaults.conf` file in your Spark install to include these lines. Replace the `username` and `password` fields with your Ocient System credentials. ```none Text theme={null} spark.sql.catalog.ocient_cat=com.ocient.spark.v2.DefaultSource spark.sql.catalog.ocient_cat.url=jdbc:ocient://host:port/db spark.sql.catalog.ocient_cat.user= spark.sql.catalog.ocient_cat.password= ``` ### Use SQL Statements After registration, the Spark connector lets you treat your Ocient System like any other Spark catalog. The connector routes SQL operations through the catalog implementation. Execute the Spark command `USE` to switch to your Ocient catalog and schema for SQL statements. In this case, use the `ocient_cat` catalog and `my_schema` schema. ```sql SQL theme={null} USE ocient_cat.my_schema; ``` Subsequent commands default to your Ocient catalog and schema, so you no longer need to reference them. This example creates a new table `my_new_table` with identifier `id`, name `name`, event timestamp `event`, and the structure of an integer and string `nested_date`. ```sql SQL theme={null} CREATE TABLE my_new_table ( id BIGINT, name VARCHAR, event_time TIMESTAMP, nested_data STRUCT ); ``` Insert a row into the new table. ```sql SQL theme={null} INSERT INTO my_new_table VALUES (1,'foo', '2025-01-01 12:00:00', (100, 'bar')); ``` Read the row from the table. ```sql SQL theme={null} SELECT * FROM my_new_table WHERE id = 1; ``` List the table. ```sql SQL theme={null} SHOW TABLES; ``` Drop the table. ```sql SQL theme={null} DROP TABLE my_new_table; ``` ### Use Scala DataFrames The Ocient Spark connector integrates directly with the Spark DataFrame API, so you can read from and write to Ocient tables using familiar Spark patterns. After you configure the Ocient catalog, you can reference fully qualified table names, and the connector handles all JDBC connectivity and type mapping. The examples in this section use [Scala](https://www.scala-lang.org/) to interact with an Ocient catalog. **Examples** **Write from Spark to Ocient** This example takes an existing Spark DataFrame `df` and writes its rows into an Ocient table `my_table`. ```javascript Scala theme={null} df.write.saveAsTable("ocient_cat.my_schema.my_table") ``` **Write from Ocient to Spark** This example reads from the Ocient table `my_table` and writes its rows into a new Spark DataFrame `df2`. ```javascript Scala theme={null} val df2 = spark.table("ocient_cat.my_schema.my_table") ``` ### Ad Hoc Usage The Ocient Spark connector supports ad‑hoc reads and writes using the Spark `.format("ocient")` method. This method is useful for brief operations, but it cannot use the Spark catalog system to create, drop, or list tables. For example, this Spark command reads an Ocient table and creates the DataFrame `df` from its contents. Substitute `jdbc_connection` with the JDBC connection string for the database, the `username` and `pwd` values for your Ocient username and password, and `my_schema` and `my_table` with the schema and table name for the table to read. ```javascript Scala theme={null} val df = spark.read .format("ocient") .option("url", "jdbc_connection") .option("user", "username") .option("password", "pwd") .option("dbtable", "my_schema.my_table") .load() ``` This command takes the DataFrame `df` and appends its contents into an Ocient table. ```javascript Scala theme={null} df.write .format("ocient") .option("url", "jdbc_connection") .option("user", "username") .option("password", "pwd") .option("dbtable", "my_schema.my_table") .mode("append") .save() ``` ## **Bulk Loading Best Practices** Use these recommended OS and Spark settings to get reliable performance and avoid inconsistent writes when using the Ocient JDBC bulk loader with Spark. For details on bulk loading, see [JDBC Bulk Loading](/jdbc-manual#jdbc-bulk-loading). ### Linux SSH Configuration Increase the SSH connection capacity on Loader Nodes: * Set `MaxStartups 1024` in the OS `sshd_config` configuration file on the `loader/SSH` endpoint hosts that accept SSH connections from the bulk loader. * Restart the SSH service to apply the updated `sshd_config` configuration. For example, on an system, run `sudo systemctl restart ssh`. ### Spark Configuration Edit the `spark-defaults.conf` configuration file to include these settings: * `spark.task.maxFailures = 1` — This configuration prevents Spark from retrying failed tasks and potentially duplicating writes. * `spark.speculation = false` — This configuration prevents Spark from launching speculative duplicate tasks that can re-run writes against Ocient. ## Configuration Options You can set specific configurations for the Ocient Spark connector through standard Spark options: * Set globally using Spark configuration: Add options to your `spark-defaults.conf` file or your cluster Spark settings (e.g., `spark.sql.catalog.ocient_cat.url=...`). * Set options per job or per operation: Use Spark (`.option()`) or command-line (`--conf`) statements to set options for one-time usage. The connector passes most of these settings through to the underlying Ocient JDBC driver as connection properties, but the system interprets a few directly by the connector to shape the generated SQL. ### Connection Options These options control how the connector establishes a JDBC connection to Ocient and identify which table or query Spark should use. All options are for both read and write operations. | **Option** | **Default** | **Description** | | -------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | None | Required. The Ocient JDBC URL.

If you do not specify the `sparkMode` setting, the connector automatically sets `sparkMode=true`. | | `user` | None | Required. The Ocient username. | | `password` | None | Required. The password of the user. | | `dbtable` | None | Required for adhoc commands using `.format("ocient")`. This option is the Ocient schema and table name (for example, `schema.table`). | | `maskPassword` | 1 | Optional. Determines whether passwords are exposed in Spark connector logs. Supported values are `0` or `1`.

If this option is set to `1`, Spark connector logs mask password fields. Otherwise, Spark connector logs include password fields. | ### Read Partitioning Options These options control how Spark splits a read into multiple partitions based on a column range, affecting parallelism and data distribution during Ocient table scans. All options are for read operations only. If you do not specify any of these options, you have only one partition. | **Option** | **Default** | **Description** | | ---------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `numPartitions` | 1 | Optional. The number of Spark partitions to create for reading. | | `partitionColumn` | None | Optional. This is a numeric, date, or timestamp column to use for partitioning the read.

If you specify the `partitionColumn `option, you can also define a value range to partition using the `lowerBound` and `upperBound` options. If you do not specify a range, the connector automatically uses the full range of values. | | `lowerBound` | None | Optional. The minimum value of the range for the `partitionColumn` option.

If you use this option, you must also include the `upperBound` option. | | `upperBound` | None | Optional. The maximum value of the range for the `partitionColumn` option.

If you use this option, you must also include the `lowerBound` option. | | `ocient.minRowsPerPartition` | 1 | Optional. Minimum target number of rows per Spark partition for the read. The connector uses this as a hint to avoid creating many tiny partitions. This option guarantees that each planned partition covers at least this many estimated rows, where possible. | ### Read Performance Options These options tune how efficiently the connector fetches rows from Ocient during reads, including JDBC fetch size and Ocient System internal parallelism. All options are for read operations only. | **Option** | **Default** | **Description** | | -------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `fetchSize` | 0 | The JDBC `fetchSize` in rows for read operations.
When you set `fetchSize = N`, the connector asks Ocient to send up to `N` rows per network fetch call, which can reduce round-trips for large result sets. The default is 0, which lets the driver choose an appropriate fetch size. This option behaves the same way as the Spark standard JDBC `fetchsize` option. For details, see the [Spark documentation](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html). | | `ocient.parallelism` | 1 | Controls Ocient internal parallel execution level for read queries. When you set `ocient.parallelism = N`, the connector appends the `USING PARALLELISM N` clause to `SELECT` SQL statements so that Ocient executes each query with `N` internal workers. This option is separate from the `numPartitions` option, which controls the number of Spark tasks that run in parallel. | ### Write Performance Options These options tune how efficiently Spark writes data to Ocient. All options apply to write operations only. | **Option** | **Default** | **Description** | | ------------------------ | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `batchsize` | `4800000`

| Controls how many rows Spark sends to Ocient in each JDBC batch during a write operation. When you set `batchsize = N`, the connector groups up to `N` rows per insertion batch, which can significantly improve write throughput for large DataFrame writes. The default is 4,800,000 rows. This option behaves the same way as the Spark standard JDBC `batchsize` option. For details, see the [Spark documentation](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html). | | `connectRetryMaxRetries` | `1` | The number of times the connector retries opening a JDBC connection on the write path after the initial TCP connect returns the `connection refused` message (typically because the SQL Node is at capacity). Set this option to `0` to disable the retry. Only a TCP `connection refused` message triggers a retry. Connect timeouts and all other connect failures stay fatal. The retry runs before the connector sends any request, so the connector never executes the same batch twice. | | `connectRetryDelayMs` | `60000` | The delay, in milliseconds, between connect-refused retries on the write path. The default is 60,000 milliseconds (60 seconds). This option has no effect when you set the `connectRetryMaxRetries` option to `0`. | The Spark connector interprets the `connectRetryMaxRetries` and `connectRetryDelayMs` options directly and does not forward them to the JDBC driver as connection properties. These options apply only to the write path and do not affect reads. ## Data Type Mapping The Ocient Spark connector supports all Ocient primitive types and complex types (such as array or tuple). When Spark creates a table, the connector writes the full Spark logical type into an Ocient `TYPE HINT` clause on each column. For example, for a column that uses an Ocient TUPLE type and maps to a Spark STRUCT type, the connector generates column DDL that includes a type hint such as `TYPE HINT 'STRUCT'`. During a read operation, the connector parses this `TYPE_HINT` field to reconstruct the original Spark schema, including nested field names. If the connector does not find the hint (e.g., for a pre-existing table), the connector maps Ocient TUPLE types to Spark STRUCT types with default field names (`_1`, `_2`, etc.). ### Data Types This table shows the Spark data types that correspond to the equivalent Ocient types. The table also lists whether each Spark type supports round-trips, meaning you can write the type from Spark to Ocient (creating the table) and then read it back into Spark while preserving the original Spark type and structure. | **Spark Type** | **Ocient Type (in** `CREATE TABLE` **SQL Statement)** | **Round-trip** | | ------------------ | ----------------------------------------------------- | ---------------------- | | `StringType` | `VARCHAR` | Yes | | `LongType` | `BIGINT` | Yes | | `IntegerType` | `INTEGER` | Yes | | `ShortType` | `SMALLINT` | Yes | | `ByteType` | `TINYINT` | Yes | | `DoubleType` | `DOUBLE` | Yes | | `FloatType` | `FLOAT` | Yes | | `DecimalType(p,s)` | `DECIMAL(p,s)` | Yes | | `BooleanType` | `BOOLEAN` | Yes | | `BinaryType` | `VARBINARY` | Yes | | `DateType` | `DATE` | Yes | | `TimestampType` | `TIMESTAMP` | Yes | | `TimestampNTZType` | `TIMESTAMP` | Yes (with `TYPE_HINT`) | | `ArrayType` | `ElementType[]` | Yes (with `TYPE_HINT`) | | `StructType` | `TUPLE<<...>>` | Yes (with `TYPE_HINT`) | | `MapType` | `TUPLE<>[]` | Yes (with `TYPE_HINT`) | **Spark 4.0 Types** When you run the connector on Spark 4.0 or later, the connector detects these additional Spark types at runtime using reflection and maps them to the corresponding Ocient types and `TYPE_HINT` values, without introducing a compile-time dependency on Spark 4.0 APIs. | **Spark 4.0 Type** | **Ocient Type (in** `CREATE TABLE` **SQL Statement)** | `TYPE_HINT` | | ----------------------- | ----------------------------------------------------- | --------------------------- | | `IntervalYearMonthType` | `INTEGER` | `SPARK_INTERVAL_YEAR_MONTH` | | `IntervalDayTimeType` | `BIGINT` | `SPARK_INTERVAL_DAY_TIME` | | `isVariantType` | `VARCHAR` | `SPARK_VARIANT` | ## Configure Logging The Ocient Spark connector logs diagnostic messages through [Apache Log4j 2](https://logging.apache.org/log4j/2.x/) under the `com.ocient.spark.v2` namespace. By default, the connector inherits the root logger level configured in your Spark environment. To change the connector log level, add the following lines to the `$SPARK_HOME/conf/log4j2.properties` file. ```none Text theme={null} logger.ocient.name = com.ocient.spark.v2 logger.ocient.level = WARN ``` Replace `WARN` with one of these supported levels: `DEBUG`, `INFO`, `WARN`, `ERROR`, or `OFF`. ## Related Links [Connect Using JDBC](/connect-using-jdbc) [JDBC Manual](/jdbc-manual) [JDBC Classes and Methods](/jdbc-classes-and-methods). *** *Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.* # Join Operations Source: https://docs.ocient.com/join-operations Understand how Ocient executes SQL JOIN operations, including hash and product joins, memory considerations, and best practices for query performance. Join operations in the System deliver high performance without requiring manual fine-tuning of queries and tables. This topic explains how the Ocient System handles joins to help you better understand performance characteristics and make informed design choices. For syntax and information about supported join types, see the [JOIN](/data-query-language-dql-statement-reference#join) reference. ## Join Optimization in Ocient Before starting any join operation, the Ocient System automatically determines the optimal plan for combining tables. The system determines the best order for joining tables to reduce processing and storage demands while increasing performance. This process includes: * **Optimizing I/O with Early Filtering** — The execution engine uses various techniques to dynamically reduce input and output (I/O) for join operations by filtering unneeded values. * **Determining join order** — The optimizer automatically reorders tables for efficiency, usually from smallest to largest in terms of row count, regardless of how you order tables in the SQL statement. * **Selecting the join strategy** — The system can override user-specified join types with more performant alternatives. For example, the system can convert an `OUTER JOIN` to an `INNER JOIN` when the result remains semantically equivalent. As a result, you generally do not need to fine-tune `JOIN` queries or configure table definitions to designate a specific table order. The optimizer ensures efficient execution plans even if you do not specify optimal table or join conditions. Join performance does not generally benefit from other table configurations for query optimization, such as , Clustering Keys, secondary indexes, or compression. ## How Ocient Joins Tables The Ocient System executes all `JOIN` operations within SQL statements as either hash or product joins: **Hash Joins** * The system uses hash joins automatically whenever your `JOIN` predicate includes at least one equality comparison (e.g., `ON a.column_1 = b.column_2`). * The system splits and shuffles data so the hash join executes in parallel. This shuffling enables great scalability and performance, even with large tables. **Product Joins** * If your `JOIN` predicate does not include an equality condition, the system uses a product join (a cartesian join with a filter). For example, the `JOIN` predicate `ON a.column_1 > b.column_2` uses a product join because it is not an equality condition. * In a product join, the system broadcasts the smaller table of the join to all compute nodes, and then compares every row according to your condition. * Product joins perform more slowly when operating on large tables, so it is generally better to use an equality comparison in your `JOIN` SQL statements when possible. ## Join Storage Considerations The performance of SQL join operations is limited by the available system memory, specifically the size of the huge pages configured on the server. During the execution of hash joins, the system typically loads the smaller table of the join into memory. **Spill to Disk** If the memory required for the join operation exceeds what is physically available, the system spills intermediate data to disk, utilizing swap or temporary files. This reliance on disk I/O can significantly degrade query performance compared to in-memory processing, but it ensures that the system can complete the query without failure. **Temporary Storage Requirements** When intermediate results or hash tables exceed available memory, the join operation adjusts to using temporary disk space approximately equal to the combined sizes of the participating tables (both the build and probe sides). **Resource Contention** Because joins are resource-intensive, heavy join workloads can affect overall system throughput and potentially impact the performance of concurrent queries, particularly in systems without strict workload management configurations. ## Join Best Practices These best practices help you design join queries that balance performance, scalability, and maintainability. While the Ocient optimizer handles most join operations automatically, applying these techniques can further reduce overhead and improve query efficiency. Many of these tips are common for OLAP database administration and reflect proven approaches across analytical systems. ### Moderate Denormalization In columnar analytical databases, moderate denormalization is a recommended best practice. This denormalization means selectively embedding low-cardinality, frequently used dimension attributes into a fact table to reduce the number of joins. For example, duplicating a product category column in a sales fact table can remove the need for a separate join to the product dimension, while still leaving high-cardinality tables normalized. Small dimension tables (less than 1 million rows) can also benefit from Global Dictionary Compression (GDC). With GDC, the system can share dictionaries across tables, improving both performance and storage efficiency. ### Query Design Considerations Beyond general join strategies, small design choices in your SQL statements can impact performance significantly. These practices help ensure that the optimizer can make efficient decisions and avoid unnecessary overhead during execution. **Use Window Aggregate Functions Instead of Self-Joins** In many cases, window aggregate functions can replace self-joins, especially for time period comparisons. **Window Aggregate Example (Most Performant)** In this example, the `LAG` function retrieves the revenue from the previous month within the same result set, allowing you to calculate month-over-month changes without requiring a self-join. ```sql SQL theme={null} SELECT month, revenue, LAG(revenue) OVER (ORDER BY month) AS prior_month_revenue, revenue - LAG(revenue) OVER (ORDER BY month) AS month_change FROM monthly_sales; ``` You can obtain the same results by using a self-join statement in a query, but this is less efficient. **Self-Join Example (Less Performant)** The query scans the `monthly_sales` table twice, once as `curr` and again as `prev`. This approach produces the same result as the window function but requires more overhead. ```sql SQL theme={null} SELECT curr.month, curr.revenue, prev.revenue AS prior_month_revenue, curr.revenue - prev.revenue AS month_change FROM monthly_sales curr JOIN monthly_sales prev ON curr.month = prev.month + INTERVAL '1' MONTH; ``` **Match Data Types for Join Keys** Always ensure that join keys share the same data type. Mismatched types force the system to apply implicit type conversions, which can slow down execution and interfere with query optimization. **Materialize Calculations Before Joins** Avoid using functions or expressions directly in join conditions. If you apply a calculation at join time (for example, casting a TIMESTAMP to a DATE), the system must evaluate that function for every row, which can result in losing the ability to use statistics or efficient hash keys. A better method is to pre-compute the value and store it as a separate column during loading or as part of a staging step. This approach reduces per-row processing costs and keeps joins predictable. **Materializing Calculations Example** Materializing calculations before joins using a common table expression separates data transformation from the join itself, so the join operates on clean, precomputed keys. This approach reduces workload during the join, enables better use of indexes and statistics, and yields simpler, more predictable query plans. ```sql SQL theme={null} WITH summarized AS ( SELECT customer_id, SUM(purchase_amount) AS total_spent FROM purchases GROUP BY customer_id ) SELECT c.region, s.total_spent FROM customers c JOIN summarized s ON c.customer_id = s.customer_id; ``` ## Related Links [Data Query Language (DQL) Statement Reference](/data-query-language-dql-statement-reference) [Window Aggregate Functions](/window-aggregate-functions) [Query Performance Optimizations](/query-performance-optimizations) # JSON Selectors Examples in Data Pipelines Source: https://docs.ocient.com/json-selectors-examples-in-data-pipelines Examples for selecting scalar values, arrays, nested fields, and NULL values from JSON records in Ocient data pipelines using field-selector expressions. The data pipeline functionality enables you to load data in the structured JSON data format. Use these examples to see how you can access the structure in different ways. For an overview of loading JSON data, see: * [Loading JSON Data](/data-formats-for-data-pipelines#load-json-data) * [Data Pipeline Load of JSON Data from Kafka](/data-pipeline-load-of-json-data-from-kafka) ## Scalar Extraction from JSON These examples load a scalar into the `VARCHAR` column named `customer_name`. Use the `CREATE TABLE` SQL statement to create the table with this column. ```sql SQL theme={null} CREATE TABLE my_schema.my_table (customer_name VARCHAR NOT NULL); ``` For a JSON file with the value at the top level, the selector can refer to the key directly. Load a file with this data. ```json JSON theme={null} {"name": "John"} ``` Use the selector `$name`. ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $name AS customer_name; ``` List all top-level JSON keys used in the `SELECT` statement in `JSON_FIELDS` of the `CREATE PIPELINE` SQL statement. For a JSON file that has a nested value, you need a complex selector. ```json JSON theme={null} {"a": {"b": {"name": "John"}}} ``` In this case, the same `CREATE PIPELINE` statement for this file has `$a.b.name` instead of `$name`. ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $a.b.name AS customer_name; ``` ## NULL and Empty Handling for JSON Scalars The System handles all JSON NULL, empty, and missing values in the same way. The system loads these values as `NULL`. These values fail to load into non-nullable columns. Provide an explicit default in the pipeline or use the `COLUMN_DEFAULT_IF_NULL` option to accept the configured column default instead of attempting to load `NULL` values. The JSON data contains this information. ```json JSON theme={null} {"a": {"b": {"name": "John"}}} {"blah": {}} ``` In this case, the output table has two rows because there are two records in the JSON source file. ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $a.b.name AS customer_name; ``` Output ```sql SQL theme={null} Ocient> select * from my_schema.my_table; name --------------------------------------------- John NULL Fetched 2 rows ``` When you load all these JSON files with the same pipeline definition, the Ocient System creates a table with the same two rows. The middle value is empty. ```json JSON theme={null} {"a": {"b": {"name": "John"}}} {"a": {"b": {}}} ``` The middle value is NULL. ```json JSON theme={null} {"a": {"b": {"name": "John"}}} {"a": {"b": null} ``` The leaf value is NULL. ```json JSON theme={null} {"a": {"b": {"name": "John"}}} {"a": {"b": {"name": null}}} ``` ## Array Extraction from JSON Supported array extraction scenarios are: * One-dimensional arrays. * Multi-dimensional arrays. * Array projection is the nested application of JSON selectors over the elements of arrays for both one-dimensional and multidimensional arrays. * Arbitrary nesting of arrays within JSON objects, including splitting array dimensions across the JSON path. * Loading individual array elements, which can also be arrays. Mapping a transformation function over array elements is not supported. ## One-Dimensional Arrays The next few examples load the same data into the `my_table` table both directly and using array projection. ```sql SQL theme={null} CREATE TABLE my_schema.my_table (customer_names VARCHAR[] NOT NULL); ``` Load data directly from a JSON array of strings. The JSON data contains this information. ```json JSON theme={null} { "a": { "b": { "names": ["John", "Bob", "Rajiv"] } } } ``` ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $a.b.names[] AS customer_names; ``` Output ```sql SQL theme={null} Ocient> select * from my_schema.my_table; customer_names -------------------------------------------------------------------------------- [John, Bob, Rajiv] Fetched 1 row ``` ## Array Projection Array projection is the application of the specified selector over all elements in an array, similar to a for loop. Load data using array projection into the same table using this JSON data. ```json JSON theme={null} { "a": [ {"name": "John", "hometown": "Chicago"}, {"name": "Bob", "hometown": "Cucamonga"}, {"name": "Rajiv", "hometown": "Glendale Heights"} ] } ``` ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $a[].name AS customer_names; ``` Output ```sql SQL theme={null} Ocient> select * from my_schema.my_table; customer_names -------------------------------------------------------------------------------- [John, Bob, Rajiv] Fetched 1 row ``` The output of both loading operations is the same. Repeat this example with data in an array that contains nested objects. The JSON file contains this information. ```json JSON theme={null} { "a": [ {"b": {"name": "John", "hometown": "Chicago"}}, {"b": {"name": "Bob", "hometown": "Cucamonga"}}, {"b": {"name": "Rajiv", "hometown": "Glendale Heights"}} ] } ``` ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $a[].b.name AS customer_names; ``` Output ```sql SQL theme={null} Ocient> select * from my_schema.my_table; customer_names -------------------------------------------------------------------------------- [John, Bob, Rajiv] Fetched 1 row ``` The output is still the same. ## Multi-Dimensional Arrays Multi-dimensional arrays work similarly to one-dimensional arrays. These examples load the same data into the `my_table` table both directly, and then using array projection. ```sql SQL theme={null} CREATE TABLE my_schema.my_table (customer_names VARCHAR[][][] NOT NULL); ``` Load data directly from a JSON array of strings. The JSON file contains this information. ```json JSON theme={null} { "a": { "b": { "names": [ [["John", "Bob", "Rajiv"], ["Anna", "Hanna", "Vanna", "Rosanna"]], [["Masha", "Natasha", "Sasha"], ["Chad", "Thad", "Brad"]]] } } } ``` ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $a.b.names[][][] AS customer_names; ``` Load data using array projection. The JSON file contains this information. ```json JSON theme={null} { "a": [ [ [ {"b": {"name": "John", "hometown": "Chicago"}}, {"b": {"name": "Bob", "hometown": "Cucamonga"}}, {"b": {"name": "Rajiv", "hometown": "Glendale Heights"}}, ], [ {"b": {"name": "Anna", "hometown": "Chicago"}}, {"b": {"name": "Hanna", "hometown": "Cucamonga"}}, {"b": {"name": "Vanna", "hometown": "Glendale Heights"}}, {"b": {"name": "Rosanna", "hometown": "Mahwah"}}, ], ], [ [ {"b": {"name": "Masha", "hometown": "Boston"}}, {"b": {"name": "Natasha", "hometown": "Cambridge"}}, {"b": {"name": "Sasha", "hometown": "Summerville"}}, ], [ {"b": {"name": "Chad", "hometown": "Champaign"}}, {"b": {"name": "Thad", "hometown": "Urbana"}}, {"b": {"name": "Brad", "hometown": "Mahomet"}}, ], ], ] } ``` `$a[][][].b.name` accesses each object, such as `{"b": {"name": "John", "hometown": "Chicago"}}`, and uses `b.name` to apply this nested attribute selector on all of the elements in the inner arrays. ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $a[][][].b.name AS customer_names; ``` Output ```sql SQL theme={null} Ocient> select * from my_schema.my_table; customer_names -------------------------------------------------------------------------------- [[[John, Bob, Rajiv], [Anna, Hanna, Vanna, Rosanna]], [[Masha, Natasha, Sasha], [Chad, Thad, Brad]]] Fetched 1 row ``` The output of both loading operations is the same. ## Multi-Dimensional Arrays With Dimensions Split Across the JSON Path To load a two-dimensional array, for example, use a one-dimensional array that contains JSON objects, each of which contains another one-dimensional array. Load the relevant parts of this JSON object into a two-dimensional array. In this example, the column has `VARCHAR[][]` type and the `SELECT` expression is `$x.a[].b.c[].name`. The loaded data is a two-dimensional array of names. The JSON file contains this information. ```json JSON theme={null} { "x": { "a": [ { "b": { "c": [ {"name": "John", "hometown": "Chicago"}, {"name": "Bob", "hometown": "Cucamonga"}, {"name": "Rajiv", "hometown": "Glendale Heights"}, ] } }, { "b": { "c": [ {"name": "Anna", "hometown": "Chicago"}, {"name": "Hanna", "hometown": "Cucamonga"}, {"name": "Vanna", "hometown": "Glendale Heights"}, {"name": "Rosanna", "hometown": "Mahwah"}, ] } }, ] } } ``` ```sql SQL theme={null} CREATE TABLE my_schema.my_table (customer_names VARCHAR[][]); ``` ```sql s theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $x.a[].b.c[].name AS customer_names; ``` Output ```sql SQL theme={null} Ocient> select * from my_schema.my_table; customer_names -------------------------------------------------------------------------------- [[John, Bob, Rajiv], [Anna, Hanna, Vanna, Rosanna]] Fetched 1 row ``` The output is a two-dimensional array of names. ## Extract Data from Individual Array Elements Extract data from an element of a JSON array. That element can also be an array. In this example, extract the first element of `a`, which is an array. JSON array indexes start at 1 so that all array indexing in Pipelines are consistent with the SQL standard. The JSON file contains this information. ```json JSON theme={null} { "a": [ {"b": [{"name": "John", "hometown": "Chicago"}]}, { "b": [ {"name": "Bob", "hometown": "Cucamonga"}, {"name": "Rajiv", "hometown": "Glendale Heights"}, ] }, {"b": [{"name": "Rajiv", "hometown": "Glendale Heights"}]}, ] } ``` ```sql SQL theme={null} CREATE TABLE my_schema.my_table (customer_names VARCHAR[]); ``` ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $a[2].b[].name AS customer_names; ``` Output ```sql SQL theme={null} Ocient> select * from my_table.my_schema; customer_names -------------------------------------------------------------------------------- [Bob, Rajiv] Fetched 1 row ``` ## NULL and Empty Handling for JSON Arrays The Ocient System handles NULL, empty, and missing values the same way for arrays as for scalars. The system converts a value that is NULL, empty, or missing to NULL and loads it as `NULL`. Provide an explicit default in the pipeline or use the `COLUMN_DEFAULT_IF_NULL` option to accept the configured column default instead of attempting to load `NULL`. ## Tuples of Scalars Data pipeline loading supports tuples of scalars in the JSON source. The specification of tuple elements using curly braces is only supported for the basic case (i.e. `$a.{b,c}`). More complex selectors are not supported. The recommended way of specifying tuples is to use an individual JSON selector for each tuple element, such as `$a.b` or `$a.c`. You can apply functions to tuple elements. ## Tuple Construction This example loads two strings into a simple tuple. ```json JSON theme={null} {"users_hometowns": {"a": {"name": "Bob", "hometown": "Cucamonga"}}} ``` ```sql SQL theme={null} CREATE TABLE my_schema.my_table (users_hometowns TUPLE<>); ``` ```sql SQL theme={null} CREATE PIPELINE my_pipeline ... SELECT tuple<>($users_hometowns.a.name, $users_hometowns.a.hometown) AS users_hometowns; ``` Output ```sql SQL theme={null} Ocient> select * from my_schema.my_table; users_hometowns -------------------------------------------------------------------------------- <> Fetched 1 row ``` ### Advanced Tuple Construction Use these examples to explore how to construct more complex tuples with nested objects and arrays. **Select Fields from a Nested Object into a Tuple** When the target fields are nested inside a named object, prefix the braces with the path to that object. Selector: ```text Text theme={null} $a.{b, c} ``` Transform equivalent: ```text Text theme={null} tuple($a.b, $a.c) ``` This data pipeline constructs a `TUPLE<>` from fields `b` and `c` inside object `a`. ```sql SQL theme={null} PREVIEW PIPELINE test_pl SOURCE INLINE '{"a": {"b": "two", "c": 2}}' EXTRACT FORMAT JSON INTO test.tgt_tbl SELECT $a.{b, c} AS tgt_col4; ``` Output ```text Text theme={null} tgt_col4 -------------------------------------------------------------------------------- ("two", 2) ``` **Select an Array of Tuples from an Array of Objects** Appending `[]` to the parent path maps the tuple selector over every element in the array, producing an array of tuples. Selector: ```text Text theme={null} $a[].{b, c} ``` Transform equivalent: ```text Text theme={null} zip_with($a[].b, $a[].c, tuple) ``` This data pipeline produces a `TUPLE<>[]` from an array of objects, each containing fields `b` and `c`. ```sql SQL theme={null} PREVIEW PIPELINE test_pl SOURCE INLINE '{"a": [{"b": 1, "c": "John"}, {"b": 2, "c": "Bob"}, {"b": 3, "c": "Rajiv"}]}' EXTRACT FORMAT JSON INTO test.tgt_tbl SELECT $a[].{b, c} AS tgt_col5; ``` Output ```text Text theme={null} tgt_col5 -------------------------------------------------------------------------------- [(1, "John"),(2, "Bob"),(3, "Rajiv")] ``` **Select a Tuple Containing an Array Element** Individual fields inside braces can have their own brackets `[]` to indicate that the field is an array. This selector produces a tuple with one element, an array, and the other a scalar. Selector: ```text Text theme={null} $a.{b[], c} ``` Transform equivalent: ```text Text theme={null} tuple($a.b[], $a.c) ``` This data pipeline constructs a `TUPLE<>` where the first element is an array of strings and the second element is an integer. ```sql SQL theme={null} PREVIEW PIPELINE test_pl SOURCE INLINE '{"a": {"b": ["John", "Bob", "Rajiv"], "c": 3}}' EXTRACT FORMAT JSON INTO test.tgt_tbl SELECT $a.{b[], c} AS tgt_col6; ``` Output ```text Text theme={null} tgt_col6 -------------------------------------------------------------------------------- ([John,Bob,Rajiv], 3) ``` **Select an Array of Tuples with an Array Element** Combining brackets `[]` on the parent path with `[]` on a child field produces an array of tuples where each tuple contains an array element and a scalar element. Selector: ```text Text theme={null} $a[].{b[], c} ``` Transform equivalent: ```text Text theme={null} zip_with($a[].b[], $a[].c, tuple) ``` This data pipeline produces a `TUPLE<>[]` from an array of objects that each contain a nested array. ```sql SQL theme={null} PREVIEW PIPELINE test_pl SOURCE INLINE '{"a": [{"b": ["John", "Bob", "Rajiv"], "c": 4}, {"b": ["Rajiv", "Bob", "John"], "c": 5}]}' EXTRACT FORMAT JSON INTO test.tgt_tbl SELECT $a[].{b[], c} AS tgt_col7; ``` Output ```text Text theme={null} tgt_col7 -------------------------------------------------------------------------------- [([John,Bob,Rajiv], 4),([Rajiv,Bob,John], 5)] ``` **Select a Tuple with Nested Tuple Elements** Nesting tuple selectors produces tuples where the elements are also tuples. Selector: ```text Text theme={null} $.{a.{b, c}, d.{e, f}} ``` Transform equivalent: ```text Text theme={null} tuple(tuple($a.b, $a.c), tuple($d.e, $d.f)) ``` This data pipeline constructs a `TUPLE<>, TUPLE<>>>` from two nested objects. ```sql SQL theme={null} PREVIEW PIPELINE test_pl SOURCE INLINE '{"a": {"b": "John", "c": 6}, "d": {"f": "Bob", "e": 7}}' EXTRACT FORMAT JSON INTO test.tgt_tbl SELECT $.{a.{b, c}, d.{e, f}} AS tgt_col8; ``` Output ```text Text theme={null} tgt_col8 -------------------------------------------------------------------------------- (("John", 6), ("Bob", 7)) ``` **Select an Array of Tuples with a Nested Tuple Element** A dot-brace group preceded by `.` (with no field name) creates an inline tuple element within an outer tuple. When you combine such an element with brackets `[]` on the parent path, this selector produces an array of tuples where one element is a tuple. Selector: ```text Text theme={null} $a[].{b, .{c[1], c[2], c[3]}} ``` Transform equivalent: ```text Text theme={null} zip_with($a[].b, zip_with($a[].c[1], $a[].c[2], $a[].c[3], tuple), tuple) ``` This data pipeline produces a `TUPLE<>>>[]` by selecting a scalar field and an inline tuple of indexed array elements. ```sql SQL theme={null} PREVIEW PIPELINE test_pl SOURCE INLINE '{"a": [{"c": ["John", "Bob", "Rajiv"], "b": 4}, {"c": ["Rajiv", "Bob", "John"], "b": 5}]}' EXTRACT FORMAT JSON INTO test.tgt_tbl SELECT $a[].{b, .{c[1], c[2], c[3]}} AS tgt_col10; ``` Output ```text Text theme={null} tgt_col10 -------------------------------------------------------------------------------- [(4, ("John", "Bob", "Rajiv")),(5, ("Rajiv", "Bob", "John"))] ``` ## Applying Functions (Transformations) To Tuple Elements In this example, load a tuple where elements have different types, `VARCHAR` and `TIMESTAMP`, and where the second element has to be converted from a JSON string into an Ocient timestamp. The JSON file contains this information. ```json JSON theme={null} {"event_timestamp": {"event": "my_event", "timestamp": "1980-02-03 15:16:17.12345"}} ``` ```sql SQL theme={null} CREATE TABLE my_schema.my_table (event_timestamp TUPLE<> NOT NULL); ``` ```sql SQL theme={null} CREATE PIPELINE my_pipeline ... SELECT tuple<>($event_timestamp.event, TO_TIMESTAMP(char($event_timestamp.timestamp), 'yyyy-MM-dd HH:mm:ss.SSSSSS')) AS event_timestamp; ``` Output ```sql SQL theme={null} Ocient> select * from my_schema.my_table; event_timestamp -------------------------------------------------------------------------------- <> Fetched 1 row ``` The output table has one row of tuples. ## NULL and Empty Handling for JSON Tuples All the rules for handling NULL, empty, and missing elements that apply to scalars and arrays also apply to tuples. If any part of the selector is NULL, empty, or missing, data pipeline loading converts that value to `NULL`. Additionally, because you can apply functions to tuple elements (and not array elements), you can use the `NULL_IF` function to convert a tuple element to `NULL`. For example, `tuple<>( $a.name, NULL_IF($a.hometown, 'N/A') )` indicates to the pipeline that the string `'N/A'` signifies `NULL` for the `hometown` element but not for the `name` element. ## Special Characters in JSON Keys This example shows loading data where the JSON key names contain a special character. Create a table for the load using the `customer_name` column. ```sql SQL theme={null} CREATE TABLE my_schema.my_table (customer_name VARCHAR NOT NULL); ``` Load a file with this data. The JSON key name contains the `-` special character. ```json JSON theme={null} {"first-name": "John"} ``` Use double quotes for the selector `$first-name` in the `CREATE PIPELINE` SQL statement. ```sql SQL theme={null} CREATE PIPELINE my_pipeline SOURCE filesystem FILTER 'data.json' EXTRACT FORMAT json INTO my_schema.my_table SELECT $"first-name" AS customer_name; ``` ## Array Extraction Operations Other operations exist in the source field selector syntax to flatten or compact arrays while extracting the data. ### Flatten Arrays Flattening means that the extracted array should not increase the rank, and instead, sub-arrays should be concatenated. To flatten an array, use the underscore character in the array selector (e.g., `$data.array[_]`). **Example** This JSON data contains a nested array with two sub-arrays having data `[1,2,3]` and `[4,5,6]`. ```json JSON theme={null} { a: { b: [ {c: [1,2,3] }, {c: [4,5,6] } ] } } ``` Flatten the nested array. ```sql SQL theme={null} a.b[_].c[] ``` Output: `[1,2,3,4,5,6]` This selected array has rank 1, and not 2. Contrast this with the non-flattened selector `a.b[].c[]`, which would have the output `[[1,2,3],[4,5,6]]`. You can also flatten arrays with the `FLATTEN` function. For details, see [Array Data Transformation Functions](/transform-data-in-data-pipelines#array-data-transformation-functions). The functionality of the operator and function is equivalent. ### Compact Arrays Compaction eliminates NULL values from the output data. The exact bit pattern of NULL in the source data is source-type dependent. For JSON, `null` is a literal keyword that is unambiguous. For CSV or other less-defined types, the configuration determines which exact bits equate to NULL. To compact an array, use the exclamation mark character in the array selector(e.g., `$data.array[!]`). **Example** This JSON data contains a nested array with four NULL values. ```json JSON theme={null} { a: { b: { c: [ 1,null,2,null,3,null,null] } } } ``` Compact the nested array. ```sql SQL theme={null} a.b.c[!] ``` Output: `[1,2,3]` You can also compact arrays with the `ARRAY_COMPACT` function. For details, see [Array Data Transformation Functions](/transform-data-in-data-pipelines). The functionality of the operator and function is equivalent. ## Related Links [Data Pipeline Load of Parquet Data from S3](/data-pipeline-load-of-parquet-data-from-s3) [Data Pipeline Load of Parquet Data from S3](/data-pipeline-load-of-parquet-data-from-s3) [Data Types for Data Pipelines](/data-types-for-data-pipelines) [Data Formats for Data Pipelines](/data-formats-for-data-pipelines) [Monitor Data Pipelines](/monitor-data-pipelines) # Key Concepts Source: https://docs.ocient.com/key-concepts Learn the core concepts behind the Ocient System, including architecture, scalability, and analytics designed for large complex data sets. The is a real-time OLAP datastore designed for running analytics against large time series data sets. The System operates in compliance with the ANSI standard for SQL. The Ocient System includes: * **Ultra-fast query speeds** for analytics on hyperscale data sets. * **Scalability** for cost and performance on a system built with commodity hardware. * **Flexible deployment** for running on premises, in the cloud, or as an Ocient-managed service. * **Robust functionality** for DDL, geospatial (), and machine learning (). * **Data movement** at scale for ETL and ELT ingestion and operations. ## The Ocient System The Ocient System is based on (CASA), which collocates NVMe drive storage with the system compute resources to optimize performance. This design keeps records near and accessible for computation without a separate storage layer, which avoids many common bottlenecks for database engines, such as limits on network capacity or processing throughput. The design achieves superior query performance when it operates on trillions of rows of data. To learn more about the design principles behind the Ocient System, see [Ocient Architecture](/ocient-architecture). ## Deployment Ocient is designed for flexible deployment for unique use scenarios that include: * — A cloud deployment managed by Ocient Support. This option uses performance-tested hardware and network infrastructure. * Public Cloud — Installation on third-party cloud providers, including and . * On premises — A self-managed option for hosting Ocient in your own data center. ## System Nodes Ocient is a distributed system consisting of interconnected nodes within clusters, which together form the data warehouse. The Ocient System designates nodes for these system roles: * **SQL Nodes** — Parse incoming SQL statements and administer commands throughout the system. These nodes serve as the interface for system and database administration. * **Loader Nodes** — Manage ETL ingestion from batch or streaming sources and index data for optimized performance. * **Foundation Nodes** — Store data and perform the bulk of query processing. Foundation Nodes perform as much query processing as possible on their stored data before interacting with other nodes. Ocient architecture diagram that shows the relationship between data sources, loading and transformation of the data, and data storage Understanding nodes and their roles in an Ocient System is most useful for system setup, administration, and maintenance. For details about nodes, see [Ocient Architecture](/ocient-architecture). However, the Ocient System makes it easy to load and query data without knowing these details. ## Data Types The OcientAIQ Unified Data Platform, the functions, and the indexes use only native SQL data types to optimize storage and performance. Supported data types include common SQL scalar types and complex types such as IP, arrays, tuples, and geospatial. For details about supported data types, see [Understanding Data Types](/understanding-data-types). ## Loading The Ocient System can load data for an end-to-end flow from file or streaming sources, including common sources such as: * AWS S3 For all the source and format options, Ocient uses pipelines to control ingestion. You can control pipelines using DDL commands, API on Loader Nodes, or a command-line interface. You can transform data during loading using SQL functions. Loading ingestion throughput can scale as needed based on the Ocient architecture. For details, see [Load Data](/load-data). ## Query Processing The design of the Ocient architecture minimizes the amount of on-disk data that must be read and processed to execute a query. To do this, the system compiles a custom I/O pipeline for each data segment relevant to a query. These custom pipelines leverage any keys or indexes to improve throughput. To learn more about query processing, see [Query Performance Optimizations](/query-performance-optimizations). ### Ocient Indexing An Ocient datastore can use multiple layers of indexing to facilitate query performance. Using these indexes is pivotal to optimizing query performance for large data sets. You can apply indexes using basic DDL statements that do not require deep knowledge of the internal system or data set. You also have some ability to customize indexes for their specific use cases. Indexes are divided into two main categories. These indexes require no additional storage and usually should be deployed on every table. Segment keys operate by accessing data from the partitioned data segments to quickly filter rows without I/O processing. Segment keys include: * **Time Key** — A segment key that partitions data based on a time-series data column. * **Clustering Key** — A series of columns that are frequently queried together. The system subdivides these segments on disk for faster reference. For more information on segment keys, see [TimeKeys and Clustering Keys](/timekeys-and-clustering-keys). When deployed precisely, Ocient secondary indexes can dramatically reduce the time for queries to run on large data sets. Ocient supports secondary indexes for these data categories: * **Numeric** * **String** * **Partial string** * **Geospatial** For more information, see [Secondary Indexes](/secondary-indexes). ## Transactions The Ocient System supports transactions that group one or more SQL statements into a single unit of work. The database either commits all changes or rolls them back as a whole. Use transactions to ensure data consistency across loading and querying operations when a set of related changes must succeed or fail together. For details, see [Transactions](/transactions). ## Connecting to Ocient Ocient provides SQL access to its database using industry-standard interfaces: * JDBC * pyocient, a -based driver for Ocient. * HTTP Query API, a REST-based interface for executing SQL statements. * Connector, a DataSourceV2 implementation for reading from and writing to Ocient tables using Apache Spark. End users can query and analyze data in Ocient without understanding the organization, nodes, networking, or other parts of the system architecture. For more information, see [Connect to Ocient](/connect-to-ocient). For details about the connection drivers, see [Connection Driver Reference](/connection-driver-reference). ### Integrations Ocient supports integration with various third-party tools for database administration, business intelligence, and system monitoring. For details about supported third-party tools, see [Ocient Integrations](/ocient-integrations). ## Machine Learning The Ocient System includes OcientML functionality for training and executing machine learning models directly within SQL. OcientML builds on native linear algebra support, including first-class matrix data types and operations such as matrix arithmetic, inversion, and eigenvalue decomposition. You can create models using the `CREATE MLMODEL` statement and execute them as scalar functions in queries. Ocient supports a broad suite of models spanning regression, classification, clustering, dimensionality reduction, ensemble methods, and neural networks to address a wide range of analytical use cases. For details, see [Machine Learning in Ocient](/machine-learning-in-ocient). ## Resilience to Hardware Failure The Ocient System uses erasure coding to organize and compute parity blocks so the system is fault-tolerant and can rebuild missing data. This failsafe does not require redundant copies of data, meaning that storage requirements are minimal. ## Security As a unified platform, Ocient helps keep data secure by consolidating security capabilities and reactions in one place with a suite of auditing and monitoring tools. Ocient supports these security standards: * **Data Encryption:** Optional through TLS/SSL protocols. * **Security Compliance:** Audited and certified SOC 2 Type 2. * **Monitoring**: Log-level monitoring and alerts on key system information. * **Access Controls**: SSO, role-based, and system-level access controls are available. ## Related Links [Core Elements of an Ocient System](/core-elements-of-an-ocient-system) [Ocient Architecture](/ocient-architecture) [Ocient Simulator](/ocient-simulator) [Connect to Ocient](/connect-to-ocient) ## Related Videos [At the Whiteboard with Ocient: Compute Adjacent Storage Architecture™](https://www.youtube.com/watch?v=7cv0hr7f1fg) [At the Whiteboard with Ocient: SQL at Scale](https://www.youtube.com/watch?v=G64g7U6tlCQ) [At the Whiteboard with Ocient: Indexing at Hyperscale](https://youtu.be/0M90mUSsiLo) # LAT Advanced Loading and Transformations Source: https://docs.ocient.com/lat-advanced-loading-and-transformations Enhance your data loading processes with advanced transformations in Ocient, tailored for complex data requirements and optimized performance. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). This guide presents examples of some more advanced transformation topics such as exploding data, loading multiple tables from a single topic, working with arrays, and accessing nested data. ## Loading Multiple Tables from a Topic The [LAT Load JSON Data from Kafka](/lat-load-json-data-from-kafka) guide shows an example of loading each topic into its own table. However, each topic can route data to multiple tables and many topics can stream load simultaneously to one or more tables in a single pipeline. Here is an example `pipeline.json` file that will load data into two separate tables: ```json JSON theme={null} { "version": 2, "pipeline_id": "pipeline-metabase", "source": { "type": "kafka", "kafka": { "bootstrap.servers": "127.0.0.1:9092", } }, "transform": { "topics": { "products": { "tables": { "metabase.public.products": { "columns": { "id": "id", "ean": "ean", "title": "title", "category": "category", "vendor": "vendor", "price": "price", "rating": "rating", "created_at": "to_timestamp(created_at, 'yyyy-MM-dd\\'T\\'HH:mm:ss[.SSS]X')" } }, "metabase.public.product_ratings": { "columns": { "created_at": "now()", "product_id": "id", "rating": "rating", } } } } } } } ``` A few key points here. 1. You are loading data from a Kafka topic called `products`. 2. That topic is loading into a table called `database.schema.table`. 3. Most columns are simply mapped from the source JSON document into, however some are transformed from the original data. 1. `created_at` is parsed from a string using the `to_timestamp` function 2. `created_at` for `product_ratings` is set dynamically to the value of the `now()` function ## Loading Nested JSON Data As you can see from the transformation, the syntax is similar to SQL, but the inner language used to traverse the JSON document and manipulate arrays can be unfamiliar. The approach used to select a property from the source JSON is to follow the tree with dot notation. For example: Given a series of two documents like: ```json JSON theme={null} { "country" : "US", "states" : ["AR", "VA", "MD", "IL"], "languages" : [ { "name" : "English", "locale" : "en-US", "dialects" : ["Northern American English", "Southeast Super-Regional English", "Western American English"] }, { "name" : "Spanish", "locale" : "es-US", "dialects" : ["Mexican Spanish", "Caribbean Spanish", "Central American Spanish", "South American Spanish", "Colonial Spanish"] } ], "capital" : { "city" : "Washington", "region" : "District of Columbia" }, "founded_in" : 1776 } { "country" : "CA", "provinces" : ["BC", "ON", "QC", "MB"], "languages" : [ { "name" : "English", "locale" : "en-CA" }, { "name" : "French", "locale" : "fr-CA" } ], "capital" : { "city" : "Ottawa", "region" : "Ontario" }, "founded_in" : 1867 } ``` You can access a property like the country simply by referencing the property: ```json JSON theme={null} /* Ocient transformations are in this format */ { "column_name" : "transformation" } /* to apply the country property to the country_code column */ { "country_code" : "country" } ``` You can access nested properties easily as well: ```json JSON theme={null} /* sets the column capital_city to values like "Ottawa" */ { "capital_city" : "capital.city" } ``` If you want to join the capital sub-properties together, you can easily do so: ```json JSON theme={null} /* sets the column capital_city to values like "Ottawa, Ontario" */ { "capital_city" : "concat(capital.city, ',', capital.region)" } ``` Note that these two documents are not fully cleansed, and you want to capture the states or provinces array and assign it to a common destination column, regions. You can do this with a coalesce operator `||`: ```json JSON theme={null} /* sets the column regions to ["AR", "VA", "MD", "IL"] for the US and ["BC", "ON", "QC", "MB"] for Canada */ { "regions" : "states || provinces" } ``` ## Loading Array Data Where arrays are encountered, special array notation is used. The benefit of this syntax is that properties in objects in arrays can also be accessed to return an array. Another useful capability is the flattening of arrays. In the event that the values returned from a transformation includes nested arrays, you can flatten it to a single dimension array by appending `[]` to the end of the transformation expression. For example: ```json JSON theme={null} /* For the US, this transformation would return a structure like: [ ["", ""], ["", ""] ] */ { "regions" : "languages.dialects" } /* You can flatten this to return this one-dimensional array with structure like: [ "", "", "", "" ] */ { "regions" : "languages.dialects[]" } ``` ## Exploding Array Data In some cases, a single record from a loading source should represent more than one row in the target table. This is referred to as exploding the data. This allows nested array data to be expanded and to be associated with the corresponding values from the record. For example, if you would like to "explode" the data in the languages array into a single row per value to populate a standalone "country\_languages" table, you could do the following for the `transform` section of your `pipeline.json` file: ```json JSON theme={null} { "transform": { "tables": { "my_database.my_schema.country_languages": { "columns": { "country_name": "country", "language_name": "EXPLODE(languages[].name)", "dialects": "EXPLODE(languages[].dialects)", "locale": "EXPLODE(languages[].locale)" } } } } } ``` This would result in the following output. Note that instead of the original 2 records, this result will produce four rows into the new database table. The "exploded" values are unique, but the records inherit their parent values like `country` from the record when the explode occurs: **Example Rows:** ```json JSON theme={null} { "country" : "US", "language_name" : "English", "dialects" : ["Northern American English", "Southeast Super-Regional English", "Western American English"], "locale": "en-US" } { "country" : "US", "language_name" : "Spanish", "dialects" : ["Mexican Spanish", "Caribbean Spanish", "Central American Spanish", "South American Spanish", "Colonial Spanish"], "locale": "es-US" } { "country" : "CA", "language_name" : "English", "dialects" : NULL, "locale" : "en-CA" } { "country" : "CA", "language_name" : "French", "dialects" : NULL, "locale" : "fr-CA" } ``` If multiple columns in the transformation include an `EXPLODE`, they results will be "zipped" together. If the arrays are different lengths, the number of rows will match the longest array. The shorter array will populate the end of its rows with NULL. This is sometimes referred to as an "Outer Explode." ## Transforming and Joining String Data One common requirement is to split a string on a token to make an array. This is accomplished using transformation functions in the LAT. Assuming a data input with a space separated string: ```json JSON theme={null} { "states": "AL AK AZ AR CA CO CT DE" } ``` You can extract an array of state values in a few ways using the `tokenize` function. Tokenize has a number of flags to control how the pattern should be interpreted. For example `'q'` indicates the pattern is a literal and `'i'` indicates case insensitive regular expression. ```json JSON theme={null} /* You can use a literal space to split and load ["AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE"] */ { "states_col" : "tokenize(states, ' ', 'q')" } /* You can use a regular expression for whitespace to split and load ["AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE"] { "states_col" : "tokenize(states, '\\s', '')" } ``` For more complex situations, you can split on any regular expression such as cases where values are wrapped with double quotes and might contain spaces. For this, you can use the expression `(?=(([^\\'\"]*[\\'\"]){2})*[^\\'\"]*$)` that leverages a zero-width positive lookahead assertion. Note that because this regex will be presented in a JSON document, certain characters must be escaped within the JSON string. For example, in JSON: ```json JSON theme={null} { "my_column" : "tokenize(states, ' (?=(([^\\'\"]*[\\'\"]){2})*[^\\'\"]*$)', '')" } ``` This would actually pass the following regular expression to the tokenize function when the backslash escape sequences are processed: ```none Text theme={null} (?=(([^'"]*['"]){2})*[^'"]*$) ``` Similarly, note that double backslashes are required to provide escape characters in the regular expression. For example, `'\\s'` would actually pass the regular expression `\s` to the tokenize function. Similar to splitting on a token, the LAT can easily join an array into a string with the `join` function. With source data that includes an array like the following: ```json JSON theme={null} { "state_names": ["Alabama", "Alaska", "Arizona", "Arkansas", "California"] } ``` You can use the join function to produce this string: ```json JSON theme={null} /* join takes a string delimiter and will produce "Alabama, Alaska, Arizona, Arkansas, California"*/ { "state_names_col": "join(', ', state_names)" } ``` ## Related Links [LAT Overview](/lat-overview) [LAT Data Types in Loading](/lat-data-types-in-loading) [LAT Advanced Topics](/lat-advanced-topics) # LAT Advanced Topics Source: https://docs.ocient.com/lat-advanced-topics Advanced topics for the Ocient Loading and Transformation (LAT) system, including dynamic schema changes, error management, and complex pipeline behaviors. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). ## Addition of an LAT Instance to the Loading Path Use these steps to add an LAT instance to the loading path: 1. Stop the LAT process on all Loader Nodes using `sudo systemctl stop lat`. 2. Copy `/opt/lat/.lat-data/pipeline/pipeline.json` from one of the Loader Nodes to the new node. 3. On the new node, execute this command: `sudo chown lat:lat /opt/lat/.lat-data/pipeline/pipeline.json` 4. On the new node, execute this command: `sudo chmod 644 /opt/lat/.lat-data/pipeline/pipeline.json` 5. Start the LAT process on all Loader Nodes using `sudo systemctl start lat`. ## Removal of an LAT Instance from the Loading Path Use these steps to remove an LAT instance from the loading path: 1. Stop the LAT process on all Loader Nodes using `sudo systemctl stop lat`. 2. Delete the `/opt/lat/.lat-data/pipeline/pipeline.json` from the chosen node. 3. Start the LAT process on all Loader Nodes using `sudo systemctl start lat`. 4. For file loading, rebalance the load across the remaining nodes. For details, see the `pipeline rebalance` command in [LAT Client Command Line Interface](/lat-client-command-line-interface#subcommands). For a load using , shut down the service on the chosen node and delete the configuration. ## Dynamic Schema Changes The loading system in Ocient is designed to support dynamic schema changes. The LAT allows data to load continuously even while database tables are altered (e.g., ADD COLUMN, DROP COLUMN). In this way dynamic changes can occur on the database and the pipeline can be updated later to begin streaming the revised data through the pipeline into the revised tables. The typical flow for updating a table is to first execute a DDL Command such as "`ALTER TABLE … ADD COLUMN …`" and then when it is completed to execute the update on the LAT Pipeline to include (or remove) the altered column in the transformation. ## Error Sink The LAT provides 2 ways to view loading, transformation, or binding errors that occur during a pipeline. ### Kafka Error Topic If a Kafka source is used, failed records can optionally be routed to a configurable [error\_topic](/lat-pipeline-configuration#error_topic). This topic will contain the records that fail and their failure reason/exception. The topic can be used to gain insight into why a given record failed to load. Each entry in the topic will contain the following: * Value\[byte\[]]: A byte array containing the record itself. * Headers: * topic: The topic from which the record originated. * partition: The partition from which the record originated. * offset: The partition offset from which the record originated. * state: The state of the record (the location where the record encountered an error). * exception: The exception associated with the error * exception\_message: The exception message, if it exists. ### Error Log File If you use a File source, or if the Error Topic configuration is not set, the LAT sends errors to a dedicated error log file. If you use the default `log4j2` configuration, this error log can be found alongside the rest of the LAT logs in `error.log` and accessed using the LAT Client with the [Subcommands](/lat-client-command-line-interface). If you use a custom `log4j2` configuration, your appender configuration should look similar to this code. ```xml XML theme={null} ``` If the [LAT\_ALLOW\_LOG\_ORIGINAL\_RECORDS](/lat-service-configuration#lat_allow_log_original_records) service configuration and the [log\_original\_records](/lat-pipeline-configuration#log_original_records) pipeline configuration are both set to `true`, the error log file includes a JSON representation of the record that caused an error. If you use a custom `log4j2` configuration, you must still use a `RollingRandomAccessFileAppender` named `ErrorLog`. Also, `PatternLayout` must still be `%m%n`, because the errors API can retrieve errors. If you do not use `RollingRandomAccessFileAppender`, the LAT does not start. If `PatternLayout` is not correct, the errors API does not work. ## Understanding Deduplication The LAT loads rows from data sources in an exactly-once fashion. This is made possible by row level deduplication for LAT pipelines and the ability to replay records from a source. In short, if the same records are replayed through the LAT, there are specific scenarios that guarantee that no duplicate records will be persisted into the Ocient tables. For a deeper understanding this works, a few key concepts are explained here: * Partitioning Data * The Durability Horizon * Deduplication Scope ### Partitioning Data To deliver high throughput loading, the LAT partitions the data source into independent sets of data. These are then loaded in parallel across all LAT instances. Each partition is considered a well-ordered sequence of rows that is replayable. Some sources like Kafka natively support the concept of partitions and have a native record ID as part of their protocol. Others, like a batch of files from S3, require LAT to partition the data on its own and assign a record ID. For file loading, LAT establishes a record ID based on the sorted list of files in the file group and the row of each record within the files. Altering the list of files in the target directory on the source system can change the record ID. This will impair the ability of the LAT to properly deduplicate rows if a file loading pipeline is stopped and restarted. ### The Durability Horizon Within a partition, each record is assigned a unique record ID. This ID is monotonically increasing within a partition. As data is loaded through the LAT and into Ocient’s page stores, this data is said to become "Durable" meaning that in the event of a node shutdown, the data would be preserved on non-volatile storage. At this point the record is no longer in memory, but stored on disk in a redundant fashion. The "Durability Horizon" is the largest record ID that has become Durable on each partition of data. If a previously loaded record were replayed it is recognized as a duplicate of the original record and ignored. New records are loaded and the Durability Horizon is increased. ### Deduplication Scope Deduplication is constrained based on a few settings in the LAT pipeline configuration. Each pipeline ID is considered an independent loading task. Additionally, each topic or file group is considered an independent data set. As a result, deduplication does not apply between different pipelines or different topics and file groups even if they are loading the same underlying files. Records are deduplicated when all of the following are true: 1. The pipeline ID matches 2. The topic name or file group name matches When this is true, any record with a record ID less than or equal to the current Durability Horizon will be considered a duplicate and ignored. When the record ID has progressed higher than the Durability Horizon, then new data will begin loading into the database. If no `pipeline_id` is set, the LAT Client will automatically assign a randomly generated ID, resulting in no deduplication across different pipelines. In the event that you want deduplication between multiple pipelines, the `pipeline_id` should be copied from the previous pipeline and explicitly set. The topic/file groups must also be the same to ensure deduplication. When updating an existing file load pipeline to select a different group of files by altering start/stop time or another filter, be sure to use a new pipeline ID (best accomplished by avoiding setting an explicit pipeline ID when creating the pipeline with the client). Otherwise, unexpected results can occur such as rows in a new file being considered a duplicate. ### V1 → V2 Migration LAT V2 modified the way the deduplication scope is calculated. In V1 only the topic name was used for scope calculation. Therefore, to maintain deduplication in a pipeline that is being upgraded from V1 to V2 **must be explicitly set to (the empty string)** upon creation. ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Client Command Line Interface Source: https://docs.ocient.com/lat-client-command-line-interface Install and use the Ocient LAT client command-line interface to create, deploy, start, stop, monitor, and manage loading and transformation pipelines. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). The LAT Client can be used to interact with a running LAT instance. It supports subcommands for interacting with pipelines, and previewing transformations. The LAT Client is distributed in the form of a wheel file. Contact Ocient Support for the wheel that corresponds to the LAT version. ### Prerequisites * Python >= 3.8 * pip3 >= 20.2.3 * If `pip install` fails, try upgrading `pip`. * Wheel Python package * `pip install wheel` ### Install It is recommended to install the wheel in a Python virtual environment to avoid conflicts with globally installed Python packages. For the install command, replace `$VERSION` with the latest version of the LAT client. Steps: 1. Create the virtual environment. `python3 -m venv venv` 2. Activate the virtual environment. `source venv/bin/activate` 3. Install the wheel. `pip install lat_client-$VERSION-py3-none-any.whl` 4. Run commands with `lat_client COMMAND ARGS` 5. When a new terminal is opened, repeat step 2 to activate the environment and gain access to the `lat_client` command. ### Usage Get help on the command line: ```shell Shell theme={null} lat_client --help lat_client --help ``` #### Common Arguments Some arguments are available on all subcommands. For convenience, most of them can also be set using an environment variable. `--no-verify` Skip certificate validation when connecting to LAT. Ignored if using http scheme | Example: | `--no-verify` | | -------- | ------------- | `--hosts (LAT_HOSTS)` One or more LAT hosts to orchestrate. Valid domain names or IP addresses can be used. | Example: | `--hosts http://192.168.0.1:8080 http://192.168.0.2:8081` | | ------------ | -------------------------------------------------------------- | | Environment: | `export LAT_HOSTS="http://10.4.0.1:8080,http://10.4.0.2:8081"` | `--oauth-domain (LAT_OAUTH_DOMAIN)` OAuth domain to use for token acquisition. | Example: | `--oauth-domain https://dev-12345678.okta.com` | | ------------ | --------------------------------------------------------- | | Environment: | `export LAT_OAUTH_DOMAIN="https://dev-12345678.okta.com"` | `--oauth-server (LAT_OAUTH_SERVER)` Okta OAuth authorization server to use for token acquisition. | Example: | `--oauth-server abcdef000ghijklm111` | | ------------ | ----------------------------------------------- | | Environment: | `export LAT_OAUTH_SERVER="abcdef000ghijklm111"` | `--client-id (LAT_CLIENT_ID)` Okta client id to use for token acquisition. | Example: | `--client-id 12345678` | | ------------ | --------------------------------- | | Environment: | `export LAT_CLIENT_ID="12345678"` | `--client-secret (LAT_CLIENT_SECRET)` Okta client secret to use for token acquisition. | Example: | `--client-secret abc123` | | ------------ | ----------------------------------- | | Environment: | `export LAT_CLIENT_SECRET="abc123"` | `--oauth-http-proxy (LAT_OAUTH_HTTP_PROXY)` HTTP proxy URL to use for token acquisition. Authentication credentials can be passed in proxy URL. | Example: | `--oauth-http-proxy http://user:pass@some.proxy.com` | | ------------ | --------------------------------------------------------------- | | Environment: | `export LAT_OAUTH_HTTP_PROXY="http://user:pass@some.proxy.com"` | #### Subcommands `pipeline create` Create a new pipeline. For most use cases, it is advisable to leave the `pipeline_id` unset when creating a pipeline. The client will set it to a random UUID to prevent deduplication across different pipelines. In the event that you want deduplication between pipelines, the `pipeline_id` should be copied from the previous pipeline and included in the new pipeline. The [LAT Transform Configuration](/lat-transform-configuration) must also be the same to ensure deduplication. When the client is used to create a pipeline with a file source, the client will make adjustments to the source configuration such that partitions are assigned evenly across nodes. First, the client will get the number of workers from the pipeline. If one is not set, it will use the minimum configured `lat.default.workers` instead. Then, it will set `partitions = workers * num_nodes`. Finally, it will set `partitions_assigned = [workers * node_index, workers * (node_index + 1) - 1]` for each node. | **Arguments:** | `--pipeline`: path to the pipeline configuration .json file | | -------------- | ----------------------------------------------------------------------- | | **Example:** | `lat_client pipeline create --pipeline /home/user/my_new_pipeline.json` |   `pipeline get` Get the configuration for an existing pipeline. If the configured pipelines are identical, print the pipeline, otherwise an explanation of the inconsistency will be provided. For pipelines with a file source, `partitions_assigned` is ignored when checking if pipelines are identical. Additionally, the client will validate that all partitions are assigned, and that no partition is assigned more than once. | Example: | `lat_client pipeline get` | | -------- | ------------------------- | `pipeline update` Update the configuration for an existing pipeline. The new pipeline can only make changes to subfields in `transform`, *except* for any `topic` / `file_group` names. All other subfields of `transform` are allowed to change, including to the `table` and `column` fields. For pipelines with a file source, partition assignments will be copied from the existing pipeline. If the pipeline was running prior to the update, successful completion of this command will automatically restart the pipeline. | **Arguments:** | `--pipeline`: path to the pipeline configuration .json file | | -------------- | ------------------------------------------------------------------------ | | **Example:** | `lat_client pipeline update --pipeline /home/user/my_new_pipeline.json ` | `pipeline delete` Delete an existing pipeline. Unless the `--force` flag is used, a pipeline must be stopped, or deletion will fail. | **Arguments:** | - `--force`: Force a pipeline to delete regardless of running status 
- `--skip-validation`: Delete a pipeline regardless of cluster consistency | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | **Example:** | lat\_client pipeline delete --skip-validation | `pipeline start` Start the configured pipeline. Prior to starting the pipeline, the pipeline start subcommand will validate that all specified hosts are configured with an identical pipeline, except for pipelines with a file source, which must have different `partitions_assigned` such that each partition is assigned exactly once across all hosts. | Example: | `lat_client pipeline start` | | -------- | --------------------------- | `pipeline stop` Stop the configured pipeline. | Example: | `lat_client pipeline stop` | | -------- | -------------------------- | `pipeline status` Retrieve status of the pipeline. The valid pipeline statuses are `STOPPED`, `RUNNING`, `COMPLETED`, and `FAILED`. When the pipeline is `FAILED`, the file statuses will remain in processing. | **Arguments:** | `--list-files`: Lists selected files in their sorted order for each file group, along with file statuses (`completed`, `processing`, `not_started`). Output is summarized to be human readable. The system truncates large file lists. | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | `--list-all-files`: Lists all selected files in their sorted order for each file group, along with file statuses (`completed`, `processing`, `not_started`). | | **Example:** | `lat_client pipeline status ` | `pipeline errors` Retrieve errors that occur while the current pipeline runs. | **Arguments:** | - `--json`: output errors as lines of JSON rather than in the default human-readable format 
- `--max-errors` MAX\_ERRORS: an upper limit on the number of errors to retrieve (default is 100) 
- `--only-records`: only show records (not error messages or other information) 
- `--only-error-messages`: only show error messages (not records or other information) 
- `--no-records`: show all information except records | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Example:** | `lat_client pipeline errors --no-records --max-errors 10 ` | `pipeline rebalance` Rebalances partitions evenly to all provided LAT Nodes. This subcommand only applies to pipelines running file sources. This command is meant to be used in the case of a node outage during a file load. LAT file loading does not support automatic partition re-balancing and manual intervention is required. The flow is as follows: 1. LAT Node goes offline. 2. LAT operator rebalances the partitions from the offline node onto the online nodes using the client. The operator should use the rebalance command and omit the offline node from the hosts argument. 3. LAT Node comes back online. 4. LAT operator rebalances partitions using the client to include all nodes including the newly online node. The operator should use the rebalance command and include all online nodes in the hosts argument. | Example: | `lat_client pipeline rebalance` | | -------- | ------------------------------- | `sink create` Create a new sink configuration. The sink configuration file for this subcommand should match the same format as the [LAT Sink Configuration](/lat-sink-configuration). For example: ```text Text theme={null} { "type": "ocient", "remotes": ["1.2.3.4:5050"]} ``` | **Arguments:** | - `--sink`: path to the sink configuration .json file 
- `--name`: Name of the sink to create 
- `--default`: set this as the default sink | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | **Example:** | `lat_client sink create --sink /home/user/my_sink_config.json --name my-sink-name-1 --default ` | `sink delete` Delete a sink configuration. The sink configuration must not be part of a created or running pipeline. | **Arguments:** | `--name`: Name of the sink to delete | | -------------- | ----------------------------------------------- | | **Example:** | `lat_client sink delete --name my-sink-name-1 ` | `sink list` List all configured sinks. | Example: | `lat_client sink list` | | -------- | ---------------------- | `sink get` Get a sink configuration by id. | **Arguments:** | `--name`: Name of the sink to get configuration for. | | -------------- | ---------------------------------------------------- | | **Example:** | `lat_client sink get --name my-sink-name-1 ` | `preview` Preview a transformation. At most one of `--transform` or `--pipeline` can be provided. If neither is provided, the host will attempt to use the transformation configured in its pipeline. You can specify the `--extract` or `--pipeline` option. If you specify none of these options, the host uses the JSON Extractor by default. For details about record and extractor types, see the [LAT Extract Configuration](/lat-extract-configuration). You must specify the `--topic` or `--file-group` option, which should match the `topic` or `file_group` key in the specified `transform` section. | **Arguments:** | - `--topic` name of the topic the records are associated with 
- `--file-group` name of the file group the records are associated with 
- `--records` path to a file of records to transform. Record formats can be of Delimited Records (e.g., CSV, TSV), JSON Records, or Fixed Width Binary Records. 
- `--extract` \[Optional] path to a .json file containing the extract section of a pipeline definition to use for extraction. 
- `--transform` \[Optional] path to a .json file containing the transform section of a pipeline definition to use for transformation. 
- `--pipeline` \[Optional] path to a .json file containing a pipeline to use for transformation and extraction, if present. | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Example without pipeline:** | `lat_client preview --topic test_topic --records ./data/my_records --extract /home/user/my_extract.json --transform /home/user/my_transform.json ` | | **Example using pipeline:** | `lat_client preview --topic test_topic --records ./data/my_records --pipeline /home/user/my_pipeline.json ` | ### Common Workflows #### Check on Status of LAT Pipelines ```shell Shell theme={null} $ lat_client pipeline status 10.4.0.1:8080: STOPPED 10.4.0.2:8080: STOPPED 10.4.0.3:8080: STOPPED $ lat_client pipeline status --hosts https://10.4.0.1:8080 10.4.0.1:8080: STOPPED ``` #### Updating an Existing LAT Pipeline ```shell Shell theme={null} lat_client pipeline update --pipeline /home/user/my_new_pipeline.json ``` If the pipeline was running prior to the update, successful completion of this command will automatically restart the pipeline. If unsuccessful, the CLI will report an error with an explanation of what is wrong with the command. Common issues are invalid JSON, missing a required column. An unsuccessful update of the pipeline config does not impact actively running pipelines. #### Restart the Pipeline ```shell Shell theme={null} ## First, stop the running pipelines. To send custom hosts, use --hosts IP1:port IP2:port $ lat_client pipeline stop Stopped: 10.4.0.1:8080, 10.4.0.2:8080, 10.4.0.3:8080 ## Then, start the pipelines. $ lat_client pipeline start Started: 10.4.0.1:8080, 10.4.0.2:8080, 10.4.0.3:8080 ``` #### Check Multiple LAT Nodes to See If the Pipeline Configurations Are Compatible ```shell Shell theme={null} ## hosts with compatible pipelines $ lat_client pipeline get --hosts https://10.4.0.1:8080 https://10.4.0.2:8080 https://10.4.0.3:8080 { "pipeline_id": "91326229-5fc6-4542-99ce-87bdcb00a978", "version": 2, "source": { ... }, "sink": { ... }, "transform": { ... } } ## hosts with incompatible pipeline $ lat_client pipeline get --hosts https://10.4.0.1:8080 https://10.4.0.2:8080 https://10.4.0.3:8080 Hosts responded with incompatible pipelines: b5a6e8647cd3a28d4a5bd07cd1c491ce: 10.4.0.1:8080, 10.4.0.2:8080 160f270e39d7a51a431c841613e4dc4e: 10.4.0.3:8080 ``` The CLI will compare the MD5 hash of the pipeline configurations on all nodes and respond that all pipelines match or are inconsistent. ### LAT Client Command Line Interface Troubleshooting If the certificate authorities on the system running the Python client (LAT Client) need to be updated, an error can occur. This type of error message might appear. ```shell Shell theme={null} ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997) ``` You can run this command to resolve the error. ```shell Shell theme={null} pip install -upgrade certifi ``` The root cause of this error can be a connection to either the LAT Server over SSL or an authentication to Okta to obtain an access token over SSL. You can run the same command in both cases to resolve the issue. When the SSL certificate is self-signed on the LAT Server, you can use the `--no-verify` flag when you connect to the LAT Server without verifying the SSL Certificate. ### Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) [Install an Ocient System](/install-an-ocient-system) # LAT Data Transformation with JMESPath Source: https://docs.ocient.com/lat-data-transformation-with-jmespath Use JMESPath expressions in Ocient LAT pipelines to transform, filter, and reshape incoming JSON data records before writing them to the data warehouse. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). Within a transformation each column is configured with a JMESPath expression describing where to extract the column’s value and any transformations to apply to that value. For information on common JMESPath functions and filtering tools, see the [Standard JMESPATH Functions](/lat-transformation-functions#standard-jmespath-functions) section. For a list of functions to modify data during loading, see the [LAT Transformation Functions](/lat-transformation-functions) page. Information on customizing your own transformation functions can be found on the [LAT User-Defined Transformations](/lat-user-defined-transformations) page. For expressions to select elements from an array matching certain criteria, see the [LAT Record Filtering](/lat-record-filtering) page. ## Notes/Warnings ### JSON Keys with Numbers and Special Characters You must escape JSON keys that begin with a number or special character in the pipeline configuration. * For a JSON record: `{ "0_123" : "value" }` transform this record as follows to write "value" to "my\_col" --- `{ "my_col" : "\"0_123\"" }` * For a JSON record: `{ "a-col" : 123 }` transform this record as follows to write "value" to "my\_col" --- `{ "my_col" : "\"a-col\"" }` ### Case Sensitivity * The Ocient System normalizes database objects (e.g., table names, column names, schemas) in LAT Transformations according to the same rules as the database unless explicitly escaped with quotes. * The Ocient System normalizes all names to lowercase unless explicitly escaped. For example, the transformation: `{ "myCol" : "value" }` normalizes to `mycol` and matches a database column `mycol` that was not escaped at creation time. * If you create the database column as an escaped literal such as `"myCol"` then the LAT transformation must be specified as: `{ "\"myCol\"" : "value" }` to correctly map to the column. * Transformation functions are case sensitive, so represent them by using all lower case letters (e.g., `st_geomfromewkb(my_column)`). ### Literals * You must escape literal values with a backtick. This applies to numbers, booleans, and empty arrays (otherwise the Ocient System interprets the value as a JSON transformation operator). Right: ```json JSON theme={null} { "column_1": "`0`", "column_2": "`1`", "column_3": "`true`", "column_4": "`[]`" } ``` Wrong: ```json JSON theme={null} { "column_1": "0", "column_2": "1", "column_3": "true", "column_4": "[]" } ``` * You can write string literals with a single quote. ```json JSON theme={null} { "column_1": "'example_value'" } ``` ### Loading Numbers in Exponential Notation When you need to load numbers from exponential notation (e.g. `1.20E-05`), you might require a conversion to load them into the target Ocient column or to perform transformations on the data. Use the `to_number` transformation to load this into a column of type `decimal` or to convert to a numeric value for further transformations like multiplication. **Example:** A source record in a CSV file might have a number like `7.00E-05` that appears as a string in the field named `source_field`. ```json JSON theme={null} { "column_1": "to_number(source_field)" } ``` This JSON code converts the string in exponential notation into the double `0.00007` that is now suitable for loading into decimal, double, or float columns, or for use in mathematical transformation functions. ### Flatten, Array Access, and Pipes For general syntax help for arrays, projections, and pipe expressions, see [JMESPath website](https://jmespath.org/specification.html). Flattening arrays, accessing entire arrays, and using pipe expressions are three important operations when you work with nested data in LAT transformations. * The flatten operator (e.g., `[]`) flattens an array one level. This also removes NULL values from the array. This can cause unintended consequences if multiple arrays need to remain aligned for data processing. * When you work with nested arrays, you can maintain array hierarchy. For example, you can replace an array index such as `*` with `[*]` to maintain the hierarchy. * You can use the pipe operator (e.g., `a.b[*] | [0]`) to stop the projection of an expression into the prior expression. ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Data Types in Loading Source: https://docs.ocient.com/lat-data-types-in-loading Reference for source and target data types supported by the Ocient Loading and Transformation (LAT) system, including conversions and edge case handling. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). After extracting from source files or topics, data can be transformed and mapped to target columns in Ocient tables. LAT automatically binds many primitive source data types to Ocient column data types and automatically converts some source types to the target column type. Different source data types automatically bind to target columns using different conventions. For example, a string in JSON such as `'123'` that loads into an `INT` column properly binds to the corresponding integer value `123`. Similarly, an integer like `456` can automatically bind to an `INT` column as `456`, a `FLOAT` column as `456.0`, and a `VARCHAR` column as `'456'`. Finally, a string including Well-Known Text (WKT) representation of a `POINT` automatically binds into an `ST_POINT` column. ### Empty String Handling When you attempt to load an empty string into a target column with LAT, most data types raise an error and the record does not load into the target table. The exceptions are `varchar`, `decimal`, `binary`, and `array` column types. In those cases, the data warehouse loads these values: * `varchar`: An empty string * `decimal`: 0.0 (with the appropriate precision) * `binary`: An empty binary blob * `array`: An empty array When you extract data from text-based formats like delimited data, the data warehouse loads empty data as an empty string unless you use the `empty_as_null` setting. ### NULL Handling If you do not provide a value for a column in the target table, this table explains what value the data warehouse stores in the column. For the purposes of default value handling, these are equivalent: * Pipeline is missing a column. * Pipeline includes a column but the value in the JSON or the source data evaluates to `null`. * Pipeline includes a column but the field is missing on the source record (e.g., `a.b` does not exist in a JSON record). To transform a NULL or missing value into a default in the pipeline, use the transformation functions. For example: `not_null(a.b, 'my_default')` | **Ocient Column Definition** | **Database Behavior When Column is Missing or Column Value Evaluates to NULL** | | ---------------------------- | ------------------------------------------------------------------------------ | | NOT NULL with default | Default | | NOT NULL with no default | Record fails | | Nullable with default | Default | | Nullable with no default | NULL | ### Data Type Binding Refer to the [Data Types](/data-types) for detailed information about each data type. These tables describe the automatic binding conventions of each data type. #### `BIGINT` | **Valid JSON Bind Types:** | Number, String | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | Number and String can be either integer or floating point values and are converted to 64-bit signed integers. Decimals are dropped. '+','-' can be used at the beginning of a string to indicate sign. String overflows result in errors, while numeric overflows are silent. | | **Example:** | \{ 
"number\_bigint": 1.0, 
"string\_bigint": "1" 
} 
-> 1, 1 | #### `BINARY` | **Valid JSON Bind Types:** | String | | -------------------------- | ---------------------------------------------------------------------------------------------------------- | | **Note:** | Converted as UTF-8 to binary: each character in the string taking its binary representation. | | **Example:** | \{ 
"string\_binary": "hello" 
} 
-> BINARY(01101000 01100101 01101100 01101100 01101111) | #### `BOOLEAN` | **Valid JSON Bind Types:** | Boolean, String | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | For a value of true, a string must be one of (case insensitive): true, t, yes, on, y, 1. 
For a value of false, a string must be one of (case insensitive): false, f, no, off, n, 0. | | **Example:** | \{ 
"boolean": true, 
"string\_0": "yes", 
"string\_1": "FALSE" 
} 
-> true, true, false | #### `DATE` | **Valid JSON Bind Types:** | String, Number | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Note:** | If String, must be in format "YYYY-MM-DD". If Number must be an EPOCH\_DAY. Use the built-in TO\_DATE function to convert a string with a different date format to an EPOCH\_DAY and bind. | | **Example:** | \{ 
"string\_date": "2021-11-04", 
"epoch\_date": 0 
} 
-> DATE(2021-11-04), DATE(1970-01-01) | #### `DECIMAL` | **Valid JSON Bind Types:** | Number, String | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Valid Logical Types:** | Decimal | | **Note:** | Number and String can be either integer or floating point values. Precision and scale for the loaded data are defined by the database table definition. The target column must support enough digits both before and after the decimal point to accommodate decimal values extracted from Fixed Width Binary or other record formats that support decimal data. If insufficient precision or scale exists in the target column, the data will fail to load. | | **Example:** | \{ 
"number\_decimal": 1.0, 
"string\_decimal": "1.1" 
} 
-> DECIMAL(1.0), DECIMAL(1.1) | #### `DOUBLE`, `DOUBLE PRECISION` | **Valid JSON Bind Types:** | Number, String | | -------------------------- | ----------------------------------------------------------------------------------------- | | **Note:** | Number and String can be either integer or floating point values. | | **Example:** | \{ 
"number\_double": 1.0, 
"string\_double": "1" 
} 
-> 1.0, 1.0 | #### `HASH` | **Valid JSON Bind Types:** | String | | -------------------------- | ----------------------------------------------------------------------------------------------------------------- | | **Note:** | Converted as hexadecimal to binary: each character in the string must be a valid hexadecimal digit /\[0-9a-f]+/i. | | **Example:** | \{ 
"hex\_string": "0a1b" 
} 
-> HASH(0000 1010 0001 1011) | #### `INT` | **Valid JSON Bind Types:** | Number, String | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | Number and String can be either integer or floating point values and are converted to 32-bit signed integers. Decimals are dropped. '+','-' can be used at the beginning of a string to indicate sign. String overflows result in errors, while numeric overflows are silent. | | **Example:** | \{ 
"number\_int": 1.0, 
"string\_int": "1" 
} 
-> 1, 1 | #### `IPV4` | **Valid JSON Bind Types:** | String | | -------------------------- | --------------------------------------------------------------------------- | | **Note:** | Source data must be in the form: \[0-255].\[0-255].\[0-255].\[0-255] | | **Example:** | \{ 
"string\_ipv4": "192.168.0.1" 
} 
-> IPV4(192.168.0.1) | #### `IP` | **Valid JSON Bind Types:** | String | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | Source string can be in IPV4 or IPV6 format. IPV6 can appear in full form as shown in this example or in condensed form. Condensed form can use :: notation (e.g., 2001:cdba::3257:9652) and leading zeros in IPV6 hextets can also be omitted (e.g., 2001:0db8:0000:0000:0000:0000:3257:9652 → 2001:db8:0:0:0:0:3257:9652). IPV6 can also appear as a string of 32 hex characters without colons (e.g., 2001cdba000000000000000032579652). | | **Example:** | \{ 
"string\_ip\_ipv4": "192.168.0.1", 
"string\_ip\_ipv6": "2001:cdba:0000:0000:0000:0000:3257:9652" 
} 
-> IP(192.168.0.1), IP(2001:cdba:0000:0000:0000:0000:3257:9652) | #### `LINESTRING`, `ST_LINESTRING` | **Valid JSON Bind Types:** | String | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | Must be in the WKT format "LINESTRING(LONG LAT, … )" or "LINESTRING EMPTY" | | **Example:** | \{ 
"string\_st\_linestring": "LINESTRING(-71.064544 42.28787, -90.444444 89.562993)" 
} 
-> ST\_LINESTRING(-71.064544 42.28787, -90.444444 89.562993) | #### `MATRIX` | **Valid JSON Bind Types:** | Array\[Array], String | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Note:** | Arrays must be two-dimensional with the same rows and columns as specified in the table. String must be in JSON two-dimensional array format. Each element will be cast according to the element type’s rules in this section. | | **Example:** | \{ 
"int\_matrix": \[\[0, 1, 2],\[3, 4, 5],\[6, 7, 8]], 
"string\_matrix": "\[\[0,1],\[2,3]]" 
} 
-> MATRIX\[\[0,1,2],\[3,4,5],\[6,7,8]], MATRIX\[\[0,1],\[2,3]] | #### `POINT`, `ST_POINT` | **Valid JSON Bind Types:** | String | | -------------------------- | ------------------------------------------------------------------------------------------------------------- | | **Note:** | Must be in the format "POINT(LONG LAT)" | | **Example:** | \{ 
"string\_st\_point": "POINT(-71.064544 42.28787)" 
} 
-> ST\_POINT(-71.064544, 42.28787) | #### `POLYGON`, `ST_POLYGON` Ensure that polygons are oriented in your intended way when you load them as polygon orientation matters in Ocient semantics. If you intend all polygons to be oriented counter-clockwise, use the [st\_forcepolygonccw](/lat-transformation-functions#st_forcepolygonccw) transform function to enforce counter-clockwise semantics. | **Valid JSON Bind Types:** | String | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | Must be in the WKT format "POLYGON((LONG LAT, … ))" or "POLYGON((LONG LAT, … ), (LONG LAT, … ))" or "POLYGON EMPTY" | | **Example:** | \{ 
"string\_st\_polygon": "POLYGON((-87.62 41.87, -89.40 43.07, -87.906 43.04, -87.62 41.87))" 
} 
-> ST\_POLYGON((-87.62 41.87, -89.40 43.07, -87.906 43.04, -87.62 41.87)) | #### `REAL`, `FLOAT`, `SINGLE PRECISION` | **Valid JSON Bind Types:** | Number, String | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | **Note:** | Number and String can be either integer or floating point values. Precision and scale are inherited from the table definition. | | **Example:** | \{ 
"number\_float": 1.0, 
"string\_float": "1" 
} 
-> 1.0, 1.0 | #### `SMALLINT` | **Valid JSON Bind Types:** | Number, String | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | Number and String can be either integer or floating point values and are converted to 16-bit signed integers. Decimals are dropped. '+','-' can be used at the beginning of a string to indicate sign. String overflows result in errors, while numeric overflows are silent. | | **Example:** | \{ 
"number\_smallint": 1.0, 
"string\_smallint": "1" 
} 
-> 1, 1 | #### `TIME` | **Valid JSON Bind Types:** | Number, String | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | String must be in the format: HH:MM:SS\[.SSSSSSSSS]. Number must represent nanoseconds since midnight. Use the TO\_TIME transformation function to convert an arbitrary string format to nanoseconds and bind. | | **Example:** | \{ 
"number\_time": 1000000000, 
"string\_time": "07:33:10.0001" 
} 
-> TIME(00:00:01.0000), TIME(07:33:10.0001) | #### `TIMESTAMP` | **Valid JSON Bind Types:** | Number, String | | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | String must be in the format: YYYY-MM-DD HH:MM:SS\[.SSSSSSSSS]. Number must represent nanoseconds since epoch. Use the TO\_TIMESTAMP transformation function to convert an arbitrary string format to nanoseconds and bind. | | **Example:** | \{ 
"number\_timestamp": 1000000000, 
"string\_timestamp": "2021-11-04 07:33:10.0001" 
} 
-> TIMESTAMP(1970-01-01 00:00:01.0000), TIMESTAMP(2021-11-04 07:33:10.0001) | #### `TINYINT`, `BYTE` | **Valid JSON Bind Types:** | Number, String | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | Number and String can be either integer or floating point values and are converted to 8-bit signed integers. Decimals are dropped. '+','-' can be used at the beginning of a string to indicate sign. String overflows result in errors, while numeric overflows are silent. | | **Example:** | \{ 
"number\_smallint": 1.0, 
"string\_smallint": "1" 
} 
-> 1, 1 | #### `TUPLE` | **Valid JSON Bind Types:** | Array, String | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | String must be in JSON array format "\[e0,e1,e2]". Each element will be cast to the tuple element type at that element’s index from the Ocient table according to the rules for that element type in this section. | | **Example:** | \{ 
"double\_int": \[1.0, 1], 
"string\_array": \["Ocient", \[0, 1]], 
"string\_tuple": "\[1.0, 1]", 
"tuple\_tuple": \[\[1.0, 1],\["Ocient", 2]] 
} 
-> TUPLE(DOUBLE, INT)(1.0, 1), TUPLE(STRING, INT\[])("Ocient", INT\[0, 1]), TUPLE(DOUBLE, INT)(1.0, 1), TUPLE(TUPLE(DOUBLE, INT), TUPLE(VARCHAR, INT)((1.0, 1), ("Ocient", 1)) | #### `TYPE[]` | **Valid JSON Bind Types:** | Array, String | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Note:** | String must be in JSON array format "\[e0,e1,e2]". Each element will be cast to the array element type from the Ocient table according to the rules for that element type in this section. | | **Example:** | \{ 
"int\_array": \[0,1,2], 
"2d\_array": \[\[0, 1, 2],\[3, 4]], 
"string\_array": "\[0,1,2]", 
"tuple\_array": \[\[0, 1.0, "Ocient"],\[1, 2.0, "rocks"]] 
} 
-> INT\[0,1,2], INT\[]\[INT\[0,1,2],INT\[3,4]], INT\[0,1,2] TUPLE(INT, FLOAT, VARCHAR)\[(0, 1.0, "Ocient"), (1, 2.0. "rocks")] | #### `UUID` | **Valid JSON Bind Types:** | String | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | **Note:** | Source data must be a valid hyphen-separated UUID. | | **Example:** | \{ 
"string\_uuid": "40e6215d-b5c6-4896-987c-f30f3678f608" 
} 
-> UUID(40e6215d-b5c6-4896-987c-f30f3678f608) | #### `VARCHAR`, `CHAR` | **Valid JSON Bind Types:** | String, Number, Boolean | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | **Note:** | Number and Boolean types are converted to their string representations. | | **Example:** | \{ 
"number\_string": 1000, 
"boolean\_string": true, 
"string\_string": "string" 
} 
-> "1000", "true", "string" | ## Load Geospatial Data Geospatial data can be loaded from a variety of formats into Ocient. The Ocient System requires special considerations when you load geospatial data into Ocient including: * Coordinate System * Supported Geospatial Loading Formats * Geospatial Type Conversion * POINT Normalization * POLYGON Normalization #### Coordinate System Ocient uses Spatial Reference System ID (SRID) 4326 for all internal geospatial data types. SRID 4326 is defined in the [WGS84 Standard](https://csrc.nist.gov/glossary/term/world_geodetic_system_1984). Before you load the source data, all data should have the SRID 4326 data type. The loading operation ignores any source data that contains the coordinate system SRID information. ### Supported Geospatial Loading Formats #### Well-Known Text The primary data format for loading geospatial data in Ocient is the Well-Known Text (WKT) format. This string format is a portable format that can represent points, lines, and polygons. String data in WKT format can automatically load into `POINT`, `LINESTRING`, and `POLYGON` Ocient column types. In addition, arrays of strings in WKT format can automatically load into array columns of a suitable geospatial data type. The Ocient System assumes raw strings are in WKT format when the system loads these strings into geospatial column types. Each `LINESTRING` or `POLYGON` value can be up to a maximum of 512 MB in size. This means a `LINESTRING` or `POLYGON` can contain approximately 32 million point values. * WKT strings are whitespace insensitive, so `POINT(10 20)` is equivalent to `POINT (10 20)`. * WKT strings are case insensitive. * Extended Well-Known Text (EWKT) is not supported. To load EWKT data, load the substring after the EWKT semicolon as the WKT format. #### Well-Known Binary and Extended Well-Known Binary The other data formats that can load into Ocient are Well-Known Binary (WKB) and Extended Well-Known Binary (EWKB) formats. The Ocient System requires an explicit transformation function for both WKB and EWKB formats when you load data and convert it into the target column type. You can use the function `st_geomfromewkb` to transform WKB and EWKB data. In both cases, the Ocient System represents the WKB or EWKB string data as hexadecimal string data. The hexadecimal string can be one of these formats: * Only hexadecimal digits: `000000000140000000000000004010000000000000` * Prefixed with `0x`: `0x000000000140000000000000004010000000000000` * Prefixed with `\x`: `\x000000000140000000000000004010000000000000` #### Geometry Conversion Ocient provides the automatic conversion of geometries from simpler types to more complex types. The conversion allows the load of a LINESTRING column with a POINT, or a POLYGON column with a POINT or LINESTRING. This capability also allows the load of arrays of POLYGON data from the source arrays of a combination of POINT, LINESTRING, and POLYGON data, which represents a geometry collection. #### POINT Normalization During loading, Ocient automatically performs normalization of POINT data into a regular format used within Ocient. The Ocient System performs the following operations on POINT data during the load: * Constrain longitude to \[-180, 180) and latitude to \[-90, 90]; wrap around invalid coordinates using correct geographical handling. * Snap points near the pole to the pole. * Set longitude of points on the pole to 0 * Remove signed zeros from coordinates, so -0 becomes 0. #### POLYGON Normalization Ocient follows a counterclockwise rotation convention to indicate the outer ring of a POLYGON. The inner ring follows a clockwise rotation. If the Ocient System loads a POLYGON with a clockwise outer ring, the system indicates that the POLYGON is outside of the given ring. In some cases, the POLYGON load can lead to unexpected results when the source data follows a different polygon rotation convention. To account for this, the LAT provides the `st_forcepolygonccw` function that forces a counterclockwise rotation of the outer polygon ring and a clockwise rotation of the inner polygon ring when applicable. #### Geospatial Transformation Functions The LAT has a select set of transformation functions to construct geospatial types and manipulate them during the load. You can find the supported geospatial transformation functions in [LAT Transformation Functions](/lat-transformation-functions). ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) [Understanding Data Types](/understanding-data-types) [Geospatial Functions](/geospatial-functions) # LAT Extract Configuration Source: https://docs.ocient.com/lat-extract-configuration Configure the extract stage of an Ocient LAT pipeline, including source readers, record parsing, format selection, and parallelism for reliable data extraction. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). An extract configuration object. This object defines the way that data is extracted from the source records. The extract definition is applied to a specific topic or file group as defined in the source. Keys are file group names or topic names and values are [Extract Settings](#extract-settings). For file based loads, this should match a file\_group\_name defined in the `source.file_groups` section. For based loads this should match a `topic` defined in Kafka. Required keys: * None - Extract defaults to JSON for all topics or file\_groups unless provided ## Example Extract In the following example, two file groups are defined. The file groups defined in the source section are referenced in the extract section to define how data is extracted from the file group. For Kafka based loads, the topic names are used in the extract section to define extraction if required. ```json JSON theme={null} { ..., "source": { "type": "s3", "endpoint": "http://some.endpoint/", "bucket": "my_bucket", "file_groups": { "delimited_user_data": { "prefix": "crm/users/", "file_matcher_syntax": "glob", "file_matcher_pattern": "**.csv", "sort_type": "lexicographic" }, "delimited_order_data": { "prefix": "crm/orders/", "file_matcher_syntax": "glob", "file_matcher_pattern": "**.json", "sort_type": "lexicographic" } } }, "extract": { "delimited_user_data": { "record_type": "delimited", "headers": ["id", "username", "last_login_at", null, "first_name", "last_name"], "field_delimiter": "|", "trim_whitespace": true, "skip_blank_lines": true, "field_optionally_enclosed_by": "\"", "null_strings": ["NULL", "N/A"] }, "delimited_order_data": { "record_type": "delimited", "num_header_lines": 1, "exclude_columns": ["user_id", "order_total"], "field_delimiter": "," } }, "transform": { ... } } ``` ## Extract Settings ### `extract..record_type` Similar to Source and Sink config, `record_type` instructs the Pipeline to instantiate a specific extraction implementation for a given type of source record such as Delimited Records (e.g., CSV, TSV), JSON Records, or Fixed Width Binary Records. | Type: | string | | --------- | ------ | | Required: | No | | Default: | json | Allowed values: * `json`: see [JSON Extract](#json-extract) for additional configuration. * `delimited`: see [Delimited Extract](#delimited-extract) for additional configuration. * `fixed_width_binary`: see [Fixed Width Binary Extract](#fixed-width-binary-extract) for additional configuration. ## Shared Extract Configuration `extract..null_strings` `null_strings` instructs the extraction process to convert a string literal such as *NA* to a `null` value in the transformation layer. This allows different null string values to be provided. `null_strings` must be an array of strings. If `trim_whitespace` is set to true and a string value is trimmed to become a `null_string`, that value converts to `null`. | Type: | string\[] | | --------- | --------- | | Required: | No | | Default: | | ### `extract..empty_as_null` The `empty_as_null` setting specifies whether empty fields should be treated as null. An empty field is defined as a string field containing no characters (its length is 0). When set to false, empty fields will be treated literally as empty strings. When set to true, empty fields will be treated as `null` in the transformation and loading process. If `trim_whitespace` is set to true and `empty_as_null` is set to true, a field that consists of only whitespace will also be converted to `null`. | Type: | boolean | | --------- | --------------------------------------------------------------------------- | | Required: | No | | Default: | True for the Delimited and FWB extract types, False for a JSON extract type | ### `extract..trim_whitespace` The `trim_whitespace` setting specifies whether whitespace should be trimmed from the beginning and end of each string field as it is extracted. | Type: | boolean | | --------- | ------- | | Required: | No | | Default: | False | ### `extract..encoding` The `encoding` setting specifies the file encoding to use. Encodings should use standard encodings defined in the [Java Internationalization Guide](https://docs.oracle.com/en/java/javase/17/intl/supported-encodings.html#GUID-187BA718-195F-4C39-B0D5-F3FDF02C7205). | Type: | string | | --------- | ------ | | Required: | No | | Default: | UTF8 | ### `extract..replace_invalid_characters` `replace_invalid_characters` defines whether invalid characters based on the encoding type should be replaced with a replacement character (i.e. `U+FFFD`). If this setting is false and invalid characters are encountered, the record will not be loaded and a record extraction error will be logged. | Type: | boolean | | --------- | ------- | | Required: | No | | Default: | False | ## JSON Extract The JSON Extract type allows LAT to extract JSON data from the source records. There are no additional configuration settings available for JSON beyond the common Extract settings. ### Example JSON Extract Configuration In this example JSON extract configuration, the `replace_invalid_characters` setting is used to convert any invalid UTF-8 characters with the UTF-8 replacement character. The file group name assigned to this file group was defined as "json\_user\_data" in the `source` section of the pipelines. When transforming this data, you can reference columns by the JSON object keys. ```json JSON theme={null} { "extract": { "json_user_data" : { "record_type": "json", "replace_invalid_characters": true } }, ... } ``` ## Delimited Extract The Delimited Extract type allows the LAT to extract delimited data from the source records. The delimiter character is configurable to support CSV, TSV, or other delimited formats like pipe and semicolon. Fixed width files are a separate extraction type. ### Notes on Delimited Files * Delimited files can have additional comment rows that are prefixed by a `#` character. These lines are not processed and will be skipped during loading. * A maximum number of 1024 columns can be loaded in a single delimited file. * You can enclose fields with an optional enclosure character. In these cases, the enclosure character is not retained in the loaded data. * You can skip a configurable number of header lines during loading to prevent column headers from appearing as loaded data. The system parses headers from the beginning of each file if you configure [num\_header\_lines](#extract-\-num_header_lines) and do not configure [headers](#extract-\-headers). The system automatically names columns if you do not configure [ num\_header\_lines](#extract-\-num_header_lines) and [headers](#extract-\-headers). The columns are named "$0", "$1", and so on consecutively instead. * You can define multiple file groups. This allows different delimited extraction settings on each file group. * The default record delimiter is a `\n` newline character. If source data differs, this should be overridden. Records with improperly escaped "quote characters" or malformed "quoting" of column data will not parse. See the [ num\_header\_lines](#extract-\-field_optionally_enclosed_by) setting for more details. * Control characters such as the NULL byte `0x0` are automatically processed as whitespace when parsing delimited records. ### Delimited Configuration ### `extract..field_delimiter` The `field_delimiter` specifies the string that should function as a column delimiter. The field\_delimiter character applies to any unquoted and unescaped text when extracting records from the source file. This can be a single character or a multi-character string delimiter. Only one field\_delimiter string is allowed per extract. | Type: | string | | --------- | ------ | | Required: | No | | Default: | , | ### `extract..record_delimiter` The `record_delimiter` specifies the string that indicates the end of a record. The record\_delimiter character applies to any unquoted and unescaped text when extracting records from the source file. This can be a single character or a two-character string delimiter. Only one `record_delimiter` string is allowed per extract. The default is a standard newline character `\n`. Depending on the file, typical settings for this delimiter are a newline `\n`, a carriage return `\r` or a carriage return and newline `\r\n`. | Type: | string; maximum 2 characters | | --------- | ---------------------------- | | Required: | No | | Default: | \n | ### `extract..field_optionally_enclosed_by` Also referred to as the "quote character," the `field_optionally_enclosed_by` settings specifies the character that is optionally used to enclose a field or column. The data between enclosure characters can contain delimiter characters. The enclosure character itself must be escaped inside of an enclosed field. Common values are `"` to enclose fields by double quotes. Because this setting defines an optional enclosure, not all fields in the record need to be enclosed. The `escape` character used to escape the enclosure character is separately configured and defaults to `"` per the RFC-4180 standard. | Type: | string; maximum 1 character | | --------- | --------------------------- | | Required: | No | | Default: | " | **Example:** In the following pipe delimited record, the double quote character can be specified so that the final column will be `this field has a | in it` when loaded into the database. If no enclosure character is specified, this record would yield four columns instead of three. ```text Text theme={null} a|b|"this field has a | in it" ``` ### `extract..escape` The `escape` settings specifies the character that is used as an escape occurrences of the enclosing character used to value so that the following character is escaped. This is typically used in enclosed columns to escape the enclosure character defined as `field_optionally_enclosed_by`. | Type: | string; maximum 1 character | | --------- | --------------------------- | | Required: | No | | Default: | " | ### `extract..num_header_lines` The `num_header_lines` setting defines the number of header lines in the source. The system reads these header lines but does not load them as data. If you do not specify [headers](#extract-\-headers), the system uses the final line within this section to name the columns in each file. Then, the system uses these names in the Transform section to refer to column values. The system names columns for each file, so the names available in the Transform section might change if the file header changes. This setting only applies to file-based source types. | Type: | int | | --------- | --- | | Required: | No | | Default: | 0 | ### `extract..headers` The `headers` setting defines the header labels associated with each column in a delimited file. This array of values is associated with the columns in order from left to right. The strings assigned to the column names are used in the Transform section to refer to column values. Any explicitly supplied headers that are an empty string `""` or a `null` literal will be skipped during extraction. This can have performance benefits if all columns are not needed. | Type: | string\[] | | --------- | --------- | | Required: | No | | Default: | null | ### `extract..include_columns` The `include_columns` setting restricts extraction to the columns with the specified names. If you do not need all columns during transformation, you might improve performance when you use the `include_columns` setting. If there are no headers and you specify this setting, the system names columns based on their original positions before other columns are filtered out. For example, if there are four columns and `$0` and `$3` are included, the system names these columns `$0` and `$3` during transformation, not `$0` and `$1`. If you specify the `exclude_columns` setting, you cannot specify this setting. | Type: | string\[] | | --------- | --------- | | Required: | No | | Default: | null | ### `extract..exclude_columns` The `exclude_columns` setting causes the LAT to not extract any of the specified columns. If you do not need all columns during transformation, you might improve performance when you use the `exclude_columns` setting. If there are no headers and you specify this setting, the system names columns based on their original positions before other columns are filtered out. For example, if there are four columns and `$1` and `$2` are excluded, the system names these columns `$0` and `$3` during transformation, not `$0` and `$1`. If you specify the `include_columns` setting, you cannot specify this setting. | Type: | string\[] | | --------- | --------- | | Required: | No | | Default: | null | ### `extract..skip_blank_lines` The `skip_blank_lines` setting specifies whether lines with only whitespace should be skipped or processed. | Type: | boolean | | --------- | ------- | | Required: | No | | Default: | False | ### `extract..skip_comment_lines` The `skip_comment_lines` setting specifies whether lines with a leading comment character should be skipped or processed. | Type: | boolean | | --------- | ------- | | Required: | No | | Default: | true | ### `extract..comment_character` The `comment_character` setting specifies which character represents the start of a comment line. | Type: | string; maximum 1 character | | --------- | --------------------------- | | Required: | No | | Default: | '#' | ## Example Delimited Extract Configuration In this example extract configuration, a pipe delimited file type with five column names is provided. The file group name assigned to this file group was defined as "delimited\_user\_data" in the `source` section of the pipelines. Whitespace will be trimmed from each field and blank lines will be skipped instead of reporting errors. Fields with special characters can be escaped by `"`. Any literal values of `NULL` or `N/A` are converted to `null`. When transforming this data, columns can be referred to by `id`, `username`, `last_login_at`, `first_name`, and `last_name`. The fourth column is listed as `null` in the header, so it will be skipped during extraction. Because `num_header_lines` is not specified, the files are assumed to have no headers. ```json JSON theme={null} { "extract": { "delimited_user_data": { "record_type": "delimited", "headers": ["id", "username", "last_login_at", null, "first_name", "last_name"], "field_delimiter": "|", "trim_whitespace": true, "skip_blank_lines": true, "include_columns": ["id", "username", "last_name"], "field_optionally_enclosed_by": "\"", "null_strings": ["NULL", "N/A"] }, ... }, ... } ``` ## Fixed Width Binary Extract The Fixed Width Binary (FWB) Extract type allows the LAT to extract data from binary source records that have a fixed number of bytes per column. FWB source records typically come from sources like mainframe systems like the series that use Cobol Copybooks to produce character, integer, floating point, and decimal data in various byte formats. The FWB Extract type requires users to specify the byte offset, width in bytes, and the type of extraction to use on the indicated bytes for each column. The FWB type also allows users to specify a record length in bytes rather than a character record delimiter. FWB files are typically extracted using `encodings` such as `cp500` and `cp037`. ### Fixed Width Binary Data Types The following data types are supported when extracting Fixed Width Binary data. All data types require specification of `offset` and `width` to locate the bytes to extract in each record. | **Data Type** | **Description** | **Supported Settings** | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | string | Represents bytes of character data following the encoding defined in the extract settings. Bytes are extracted into a string for transformation and loading. | `encoding` - inherited | | integer | Represents a signed integer. Uses the `endianness` setting to determine how bytes should be interpreted. The `integer` type can be used for 1, 2, 4, or 8 bytes of data. These correspond to mainframe concepts of a Byte, Half-word, Full-word, and Double-word. | `endianness` - inherited | | unsigned\_integer | Represents a unsigned integer. Uses the `endianness` setting to determine how bytes should be interpreted. Can be used for 1, 2, 4, or 8 bytes of data. The largest integer data type in the Ocient System is a signed BIGINT, so the maximum value for an `unsigned_integer` is 2^63-1. | `endianness` - inherited | | float | Represents a floating point number as defined by the -754 floating point specification. Can be used for 4 or 8 bytes of data. 4 bytes represents a single precision floating point number, and 8 bytes represents a double precision floating point number. 
| | | packed\_decimal | A Packed Binary Coded Decimal format for decimal precision that packs two integral values in each byte and encodes the sign in the final byte. The number of decimal points represented by the data is supplied with a separate extraction setting. The sign is encoded in the 4 least significant bits of the final byte. A value of `0xD` or `0xB` indicates a negative sign. Any other value including the common values of `0xF`, `0xE`, `0xC` or `0xA` indicating positive or unsigned. | `endianness` - inherited 
`decimals` - number of digits to place the right of the decimal point 
`sign_format` - defaults to 'trailing' byte format indicating that the sign is encoded in the most significant four bits of the final byte; provided for future extensibility. | | zoned\_decimal | A Zoned Binary Coded Decimal format for decimal precision that packs two integral values in each byte and encodes the sign in the final byte. The number of decimal points represented by the data is supplied with a separate extraction setting. The sign is encoded in the 4 most significant bits of the final byte. A value of `0xD`, `0xB`, `0x7` indicates a negative sign. Values of `0xF`, `0xE`, `0xC` or `0xA` indicate positive or unsigned. | `endianness` - inherited 
`decimals` - number of digits to place the right of the decimal point 
`sign_format` - defaults to 'trailing' byte format indicating that the sign is encoded in the most significant four bits of the final byte; provided for future extensibility. | ## Fixed Width Binary Configuration ### `extract..record_width` The `record_width` setting specifies the total number of bytes in each record. Files are read serially, and the next record is assumed to start on the following byte. The `record_width` can be greater than the number of bytes used in extracting the individual columns. | Type: | integer | | --------- | ------- | | Required: | Yes | | Default: | | ### `extract..padding_character` The `padding_character` setting specifies a character in a fixed width file type that should be removed from extracted character data. After extracting a character field, any leading or trailing occurrences of this character will be stripped from the ends of the character data. Typical examples are the space character or an underscore. | Type: | string | | --------- | ------------------------------- | | Required: | No | | Default: | " " (in the specified encoding) | ### `extract..endianness` The `endianness` setting specifies whether the extraction of numeric fields should use Big Endian or Little Endian byte ordering to process extracted bytes. `endianness` impacts integer, packed\_decimal, and zoned\_decimal column types. | Type: | string | | --------- | ------ | | Required: | No | | Default: | big | Allowed values: * `big`: Big Endian * `little`: Little Endian `extract..columns` A extract columns configuration object. A collection of columns and their associated extraction configuration. Keys are column names and values are [Column Extracts configuration](#column-extracts). Required keys: * [extract.columns.\](#extract-\-columns-\) ## Column Extracts Column extract configurations for Fixed Width Binary records. ### `extract..columns.` A column extract configuration object keyed by a column name. A column’s value is defined as an extract definition. The extract definition will indicate the offset and width to locate the bytes to extract, the data type to use in the extraction, and any additional parameters about the bytes to extract. Required keys: * offset * width * data\_type Optional keys: * decimals * sign\_format ### `extract..columns..offset` Number of bytes to skip from the beginning of the record before beginning to extract bytes for this column. Note that `offset` is relative to the beginning of the record, so arbitrary offsets can be used including overlapping sets of bytes with other columns. | Type: | integer | | --------- | ------- | | Required: | Yes | | Default: | | ### `extract..columns..width` Number of bytes to extract from the `offset` location. | Type: | integer | | --------- | ------- | | Required: | Yes | | Default: | | ### `extract..columns..data_type` Data type to extract from the bytes that are specified by `offset` and `width`. See [Fixed Width Binary Data Types](#fixed-width-binary-data-types) for reference on available data types. | Type: | string | | --------- | ------ | | Required: | Yes | | Default: | string | Allowed values: * `string` * `integer` * `unsigned_integer` * `float` * `packed_decimal` * `zoned_decimal` Note that each `data_type` has specific settings that are allowed to control the extraction to that `data_type`. Some of these are inherited from the overall extract configuration, while others are defined on each column. ### `extract..columns..decimals` The number of numeric values that are placed to the right of the decimal point in a `zoned_decimal` or `packed_decimal` data type. For example, if `decimals` is set to `` 2` `` and the data in a `zoned_decimal` represents the numerals `12345`, this would create the decimal `` 123.45` `` in the extracted value. `decimals` configures the `scale` of the extracted decimal and the total number of digits present in the source data determines the precision. If insufficient scale or precision exists in the target Ocient column, data will fail to load. | Type: | integer | | --------- | ------- | | Required: | No | | Default: | 0 | Applies to: * `packed_decimal` * `zoned_decimal` ### `extract..columns..sign_format` In decimal types, the sign for the number can be encoded in different ways. Currently, only the `trailing` format is supported which requires the sign to be encoded in 4 bits of the last byte in the column data. This setting is provided for future extensibility. The values that indicate a positive or negative sign are listed in [Fixed Width Binary Data Types](#fixed-width-binary-data-types). | Type: | string | | --------- | -------- | | Required: | No | | Default: | trailing | Allowed values: * `trailing` - the sign is encoded in the most significant 4 bits of the last byte in the binary coded decimal. Applies to: * `packed_decimal` * `zoned_decimal` ## Fixed Width Binary Extract Example The following example illustrates extracting fixed width binary data from a file group that was given the name "order\_files" in the `source` section of the pipeline. The file has 3100 bytes per record and is only extracting a few columns. ```json JSON theme={null} { "extract": { "order_files": { "record_type": "fixed_width_binary", "record_width": 3100, "endianness": "big", "columns": { "order_id": { "offset": 0, "width": 4, "data_type": "integer" }, "username": { "offset": 4, "width": 20, "data_type": "string" }, "order_total": { "offset": 24, "width": 10, "data_type": "packed_decimal", "decimals": 2 }, "sales_tax_total": { "offset": 34, "width": 8, "data_type": "zoned_decimal", "decimals": 3 } } } } } ``` ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Load CSV Data from S3 Source: https://docs.ocient.com/lat-load-csv-data-from-s3 Learn to load CSV data from Amazon S3 directly into Ocient’s high-performance data warehouse, ensuring smooth and scalable data integration. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). A common setup for batch loading files into Ocient is to load from a bucket on S3 with time partitioned data. In many instances, a batch load is performed on a recurring basis to load new files. The LAT transforms each document into rows in one or more different tables. Ocient’s Loading and Transformation capabilities use a simple SQL-like syntax for transforming data. This tutorial will guide users through a simple example load using a small set of data in CSV format. The data in this example is created from a test set for the Business Intelligence tool. ## Prerequisites This tutorial assumes that: 1. The Ocient System has network access to S3 from the Loader Nodes. 2. An Ocient System is installed and configured with an active storage cluster (See the [Ocient Application Configuration](/ocient-application-configuration) guide). 3. Loading and Transformation is installed on the Loader Nodes. 4. A default "sink" for the Ocient Loader Nodes is configured on the system. 5. The [LAT Client Command Line Interface](/lat-client-command-line-interface) is installed. ## Step 1: Create a New Database To begin, load two example tables in a database. First, connect to a SQL Node using the [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). Then run the following DDL command: ```sql SQL theme={null} CREATE DATABASE metabase; ``` ## Step 2: Create Tables To create tables in the new database, first connect to that database (e.g., `connect to jdbc:ocient://sql-node:4050/metabase`), then run the following DDL commands: ```sql SQL theme={null} CREATE TABLE public.orders ( created_at TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, id INT NOT NULL, user_id INT NOT NULL, product_id INT NOT NULL, subtotal DOUBLE, tax DOUBLE, total DOUBLE, discount DOUBLE, quantity INT, CLUSTERING INDEX idx01 (user_id, product_id) ); CREATE TABLE public.products( created_at TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, id INT NOT NULL, ean VARCHAR(255), title VARCHAR(255), category VARCHAR(255) COMPRESSION GDC(2) NOT NULL, vendor VARCHAR(255), price DOUBLE, rating DOUBLE, CLUSTERING INDEX idx01 (category) ); ``` Now, the database tables are created and you can begin loading data. ## Step 3: Create a Data Pipeline Data pipelines are created using a simple loading configuration that is submitted to the Transformation Nodes to start loading. File Groups designate a batch of files to load. Each File Group is routed to one or more Ocient tables, and each column is the result of a transformation applied to the source document. First, inspect the data that you plan to load. Each document has a format similar to the following example: ```Text Text theme={null} /* orders */ id,user_id,product_id,subtotal,tax,total,discount,created_at,quantity 1,1,14,37.65,2.07,39.72,null,2019-02-11T21:40:27.892Z,2 2,1,123,110.93,6.1,117.03,null,2018-05-15T08:04:04.580Z,3 3,1,105,52.72,2.9,49.2,6.42,2019-12-06T22:22:48.544Z,2 ... /* products */ id,ean,title,category,vendor,price,rating,created_at 1,1018947080336,Rustic Paper Wallet,Gizmo,"Swaniawski, Casper and Hilll",29.46,4.6,2017-07-19T19:44:56.582Z 2,7663515285824,Small Marble Shoes,Doohickey,Balistreri-Ankunding,70.08,0,2019-04-11T08:49:35.932Z 3,4966277046676,Synergistic Granite Chair,Doohickey,"Murray, Watsica and Wunsch",35.39,4,2018-09-08T22:03:20.239Z ... ``` This is similar to the target schema created in Step 2, but it will require some transformation. Most transformations are identical to functions already in Ocient’s SQL dialect. To route data to the tables, you need to create a pipeline.json file that has the following structure: ```json JSON theme={null} { "version": 2, "workers": 4, "source": { "compression": "none", "type": "s3", "endpoint": "https://s3.us-east-1.amazonaws.com", "bucket": "ocient-docs", "file_groups": { "orders": { "prefix": "metabase_samples/csv", "file_matcher_syntax": "glob", "file_matcher_pattern": "**orders*.csv", "sort_type": "lexicographic" }, "products": { "prefix": "metabase_samples/csv", "file_matcher_syntax": "glob", "file_matcher_pattern": "**products*.csv", "sort_type": "lexicographic" } } }, "extract": { "orders": { "record_type": "delimited", "headers": [ "id", "user_id", "product_id", "subtotal", "tax", "total", "discount", "created_at", "quantity" ], "null_strings": [ "NULL", "N/A" ], "record_delimiter": "\r\n", "num_header_lines": 1 }, "products": { "record_type": "delimited", "headers": [ "id", "ean", "title", "category", "vendor", "price", "rating", "created_at" ], "record_delimiter": "\r\n", "num_header_lines": 1 } }, "transform": { "file_groups": { "orders": { "tables": { "metabase.public.orders": { "columns": { "id": "id", "user_id": "user_id", "product_id": "product_id", "subtotal": "subtotal", "tax": "tax", "total": "total", "discount": "discount", "created_at": "to_timestamp(created_at, 'yyyy-MM-dd\\'T\\'HH:mm:ss[.SSS]X')", "quantity": "quantity" } } } }, "products": { "tables": { "metabase.public.products": { "columns": { "id": "id", "ean": "ean", "title": "title", "category": "category", "vendor": "vendor", "price": "price", "rating": "rating", "created_at": "to_timestamp(created_at, 'yyyy-MM-dd\\'T\\'HH:mm:ss[.SSS]X')" } } } } } } } ``` The two interesting parts of this pipeline.json file are the way the file groups and the extraction settings are defined. First, note that each file group sets the S3 endpoint, a bucket, a prefix used for filtering the considered files, and then a file matcher. In this example, there is only a single file, but if there were many files matching the pattern `**orders*.csv` then they would all be part of the file group. You do not need S3 credentials because this is a public bucket, but if it were private, you can supply credentials in a few different ways. Next, note the extract section. Here, the example specifies a delimited format. If it were compressed (e.g., gzip), you could specify compression. The example also specifies the headers to associate with each column in the CSV data. Delimited extracts can specify different record delimiters (e.g., `\n`), specify field delimiters (e.g., `|`, `\t`, `,`), define how to handle empty fields or to trim whitespace, and specify strings that should be considered `NULL`. While not used in this file, an example of null strings is provided that would turn the string literal "NULL" or "N/A" into a database `NULL`. The final parameter supplied in the example is the sort type for the file load. This informs the LAT how you would like data to be ordered when loading. The ideal sort organizes files in time order according to the defined . This makes more efficient segments and is much faster to load. This example uses the lexicographic sort which orders according to the characters in the filename. Other sort types are available to use file modified time or to extract the timestamp for sorting from the file path or filename. ## Step 4: Using the Loading and Transformation CLI With a pipeline.json file ready to go, you can test this pipeline. To test, use the LAT CLI. For these examples, assume that two LATs are configured and set using an environment variable. First, configure the LAT CLI to use the hosts of the Ocient Loading and Transformation service. You can add these to every CLI command as a flag, but for simplicity you can also set them as environment variables. From a command line, run the following command replacing the IP addresses with the IP addresses of your LAT processes: ```shell Shell theme={null} export LAT_HOSTS="https://10.0.0.1:8443,https://10.0.0.2:8443" ``` If your LAT is running without TLS configured, replace the port number of your LAT Hosts with 8080 and the protocol with `http://`. Next, check on the status of the LAT: ```shell Shell theme={null} lat_client pipeline status ``` **Example response:** ```bash Bash theme={null} 10.0.0.1:8443: Stopped 10.0.0.2:8443: Stopped ``` Success! This confirms that you can reach the LAT from your CLI. If the status is "Running" it means a pipeline is already executing a pipeline. In the next step, you will update and start the new pipeline. This example uses secure connections. If you receive an SSL Error when testing, your service cannot be configured to use TLS or you might need to use the `--no-verify` flag if certificate validation fails. ## Step 5: Test the Transformation The CLI supports previewing a transformation with an example document and the pipeline file. This makes it easy to test your transformations. First, save an example document to your file system to use for this test. For this demo, you can download an example file from [https://ocient-docs.s3.amazonaws.com/metabase\_samples/csv/orders.csv](https://ocient-docs.s3.amazonaws.com/metabase_samples/csv/orders.csv) and save it to `~/orders.csv`. Next, make sure the pipeline.json file that you created is stored at `~/pipeline.json`. Now that both files are available, you can run the CLI to preview the results. Pass the preview command the topic name, the pipeline file, and the sample record file. The response contains the transformed data tied to the destination table and a list of any error records. Similar to how you can preview records on a topic for file loads, you can supply any one `file_groups` created in the extract section to preview the transformations. ```shell Shell theme={null} lat_client preview --topic orders --pipeline ~/pipeline.json --records ~/orders.csv ``` **Example response:** ```json JSON theme={null} { "tableRecords": { "metabase.public.orders": [ { "id": 1, "user_id": 1, "product_id": 14, "subtotal": 37.65, "tax": 2.07, "total": 39.72, "discount": null, "created_at": 1549921227892000000, "quantity": 2 }, { "id": 2, "user_id": 1, "product_id": 123, "subtotal": 110.93, "tax": 6.1, "total": 117.03, "discount": null, "created_at": 1526371444580000000, "quantity": 3 }, { "id": 3, "user_id": 1, "product_id": 105, "subtotal": 52.72, "tax": 2.9, "total": 49.2, "discount": 6.42, "created_at": 1575670968544000000, "quantity": 2 } ] }, "recordErrors": [] } ``` You can see that the data is transformed and the columns to which each transformed value will be mapped. If there are issues in the values, these will appear in the `recordErrors` object. You can quickly update the pipeline.json file and preview again. Now, you can inspect different documents to confirm that various states of data cleanliness like missing columns, null values, and special characters are well handled by your transformations. ## Step 6: Configure and Start the Data Pipeline With a tested transformation, the next step is to setup and start the data pipeline. First, configure the pipeline using the `pipeline create` command. This validates and creates the pipeline, but will not take effect until you start the pipeline: ```shell Shell theme={null} lat_client pipeline create --pipeline ~/pipeline.json ``` **Example response:** ```bash Bash theme={null} 10.0.0.1:8443: Created 10.0.0.2:8443: Created ``` In cases where there is an existing pipeline operating, it is necessary to stop the pipeline and remove the original pipeline before creating and starting the new pipeline. Now that the pipeline has been created on all LAT Nodes, you can start the LAT by running the `pipeline start` commands: ```shell Shell theme={null} lat_client pipeline start ``` **Example responses:** ```bash Bash theme={null} 10.0.0.1:8443: Running 10.0.0.2:8443: Running ``` ## Step 7: Confirm that Loading is Operating Correctly With your pipeline in place and running, data will immediately begin loading from the S3 file groups that you defined. If there were many files per file group, the LAT would first sort the files, then partition them for the fastest loading based on the sorting criteria you provided. ### Observing Loading Progress With the pipeline running, data immediately begins to load into Ocient. To observe this progress, you can use the `pipeline status` command from the LAT Client or monitor the LAT metrics endpoint of the Loader Nodes. You can check the status with this command by using the `--list-files` flag to include a summary of the files included in the load. ```shell Shell theme={null} lat_client pipeline status --list-files ``` **Example responses:** ```bash Bash theme={null} 10.0.0.1:8443: Running 10.0.0.2:8443: Running Pipeline Files Processed: 0 Pipeline Error Count: 0 Pipeline Files Remaining: 2 orders Files Processed: 0 Error Count: 0 Files Remaining: 1 products Files Processed: 0 Error Count: 0 Files Remaining: 1 orders Status Filename processing metabase_samples/csv/orders.csv In Process Files: processing metabase_samples/csv/orders.csv products Status Filename processing metabase_samples/csv/products.csv In Process Files: processing metabase_samples/csv/products.csv ``` You can monitor the LAT metrics endpoint manually at the command line. Or, you can use a tool like to retrieve metrics. For this example, run the curl command against the endpoint and review the result. For details on metrics, see the [LAT Metrics](/lat-metrics) Documentation. **Command:** ```curl CURL theme={null} curl https://127.0.0.1:8443/v2/metrics/lat:type=pipeline ``` If your LAT is running without TLS configured, replace the port number of your LAT Hosts with 8080 and the protocol with `http://`. **Example response:** ```json JSON theme={null} { "request": { "mbean": "lat:type=pipeline", "type": "read" }, "value": { "partitions": [ { "offsets_durable": 1, "pushes_errors": 0, "pushes_attempts": 1, "rows_pushed": 1, "offsets_written": 18759, "records_buffered": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "source_bytes_buffered": 0, "records_errors_transformation": 0, "offsets_processed": 18759, "partition": "table_orders-0", "records_filter_accepted": 1, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "offsets_pushed": 18759, "pushes_unacknowledged": 0, "invalid_state": 0, "bytes_pushed": 88, "records_errors_total": 0, "offsets_buffered": 18759, "complete": 0, "offsets_end": 1, "producer_send_errors": 0 }, { "offsets_durable": 1, "pushes_errors": 0, "pushes_attempts": 1, "rows_pushed": 1, "offsets_written": 199, "records_buffered": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "source_bytes_buffered": 0, "records_errors_transformation": 0, "offsets_processed": 199, "partition": "table_products-0", "records_filter_accepted": 1, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "offsets_pushed": 199, "pushes_unacknowledged": 0, "invalid_state": 0, "bytes_pushed": 145, "records_errors_total": 0, "offsets_buffered": 199, "complete": 0, "offsets_end": 1, "producer_send_errors": 0 } ], "paused": 1, "bytes_buffered": 0, "workers": 20 }, "timestamp": 1626970368, "status": 200, } ``` ### Check Row Counts in Tables To confirm that you are seeing results in the target tables, you can also run some simple queries to check row counts. Depending on the streamloader role settings, the time for records to become queryable can vary from a few seconds to minutes: **Example Queries:** ```sql SQL theme={null} Ocient> SELECT count(*) FROM public.orders; COUNT(*) --------------------- 18760 ``` ```sql SQL theme={null} Ocient> SELECT count() FROM public.products; COUNT() -------------------- 200 ``` Now you can explore the data in these four tables with any Ocient SQL queries. ### Check Errors In this example, all rows load successfully. However, a successful load does not always happen, and you can inspect errors using the LAT Client. Whenever the LAT process fails to parse a file correctly or fails to transform or load a record, the LAT process records an error. The LAT Client includes the `lat_client pipeline errors` command that reports the latest errors on the pipeline. A full error log is available on the Loader Nodes. These logs report all bad records and the reason that the load fails. When you load a pipeline from Kafka, the load might route errors to an error topic on the Kafka broker instead of the logs. The LAT Client does not contain the errors sent to the error topic. You can inspect these errors with Kafka utilities instead. This LAT Client command displays a maximum of 100 error messages. ```bash theme={null} theme={null} lat_client pipeline errors --max-errors 100 --only-error-messages |--------------------------------------------------| | exception_message | |--------------------------------------------------| | Column name: time1. Message: Failed to evaluate | | expression. Cause: | | java.time.format.DateTimeParseException | |--------------------------------------------------| | Column name: time1. Message: Failed to evaluate | | expression. Cause: | | java.time.format.DateTimeParseException | |--------------------------------------------------| ``` The errors indicate that there is an issue parsing the `time1` column. Options exist on the `pipeline errors` command to return JSON and to restrict the response to specific components of the error detail that includes a reference to the source location of this record. The following command returns JSON that is delimited with newline characters. You can pass the JSON output to `jq` or a file. The JSON includes the source topic or file group, the filename where the error occurred, the offset that indicates the line number or Kafka offset, and the exception message that aids in troubleshooting and identifying the incorrect record in the source data. You can use the `log_original_message` pipeline setting to provide direct access to the parsed source record for errors when appropriate. ```bash theme={null} theme={null} lat_client pipeline errors --max-errors 100 --json {"time": "2022-05-17T16:53:50.387386+00:00", "topic": "calcs", "partition": 0, "state": "TRANSFORMATION_ERROR", "exception_message": "Column name: time1. Message: Failed to evaluate expression. Cause: java.time.format.DateTimeParseException: Cannot parse time \"19:36:22\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\njava.time.format.DateTimeParseException: Cannot parse time \"19:36:22\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\nCannot parse time \"19:36:22\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"", "offset": 0, "record": null, "metadata": {"size": "3321", "filename": "calcs/csv/calcs_01.csv"}} {"time": "2022-05-17T16:53:50.404684+00:00", "topic": "calcs", "partition": 0, "state": "TRANSFORMATION_ERROR", "exception_message": "Column name: time1. Message: Failed to evaluate expression. Cause: java.time.format.DateTimeParseException: Cannot parse time \"02:05:25\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\njava.time.format.DateTimeParseException: Cannot parse time \"02:05:25\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\nCannot parse time \"02:05:25\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"", "offset": 1, "record": null, "metadata": {"size": "3321", "filename": "calcs/csv/calcs_01.csv"}} ``` ## Related Links [LAT Overview](/lat-overview) [LAT Data Types in Loading](/lat-data-types-in-loading) [LAT Advanced Topics](/lat-advanced-topics) # LAT Load JSON Data from Kafka Source: https://docs.ocient.com/lat-load-json-data-from-kafka Ingest JSON data from Apache Kafka into Ocient with a LAT pipeline. This tutorial covers source, transform, and sink configuration plus monitoring options. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). A common setup for streaming data into Ocient is to send JSON documents to and then transform each document into rows in one or more different tables. Ocient’s Loading and Transformation capabilities use a simple SQL-like syntax for transforming data. This tutorial will guide users through a simple example load using a small set of data in JSON format. The data in this example is created from a test set for the Business Intelligence tool. ## Prerequisites This tutorial assumes that: 1. A Kafka cluster is operational and can be reached by the Ocient Loader Nodes. 2. An Ocient System is installed and configured with an active storage cluster (See the [Ocient Application Configuration](/ocient-application-configuration) guide). 3. The Ocient Loader Nodes are running the latest Loading and Transformation version which is configured to connect to Kafka for stream loading. 4. A default "sink" for the Ocient Loader Nodes is configured on the system. 5. The [LAT Client Command Line Interface](/lat-client-command-line-interface) is installed. 6. The test data for this Tutorial can be found at the following S3 addresses. You must be logged into to download these files. `https://ocient-docs.s3.amazonaws.com/metabase_samples/jsonl/orders.jsonl` `https://ocient-docs.s3.amazonaws.com/metabase_samples/jsonl/products.jsonl` `https://ocient-docs.s3.amazonaws.com/metabase_samples/jsonl/people.jsonl` `https://ocient-docs.s3.amazonaws.com/metabase_samples/jsonl/reviews.jsonl` ## Step 1: Create a New Database To begin, you are going to load four example tables in a database. First, connect to a SQL Node using the [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). Then run the following DDL command: ```sql SQL theme={null} CREATE DATABASE metabase; ``` ## Step 2: Create Tables To create tables in the new database, first connect to that database (e.g., `connect to jdbc:ocient://sql-node:4050/metabase`), then run the following DDL commands: ```sql SQL theme={null} CREATE TABLE public.orders ( created_at TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, id INT NOT NULL, user_id INT NOT NULL, product_id INT NOT NULL, subtotal DOUBLE, tax DOUBLE, total DOUBLE, discount DOUBLE, quantity INT, CLUSTERING INDEX idx01 (user_id, product_id) ); CREATE TABLE public.people ( created_at TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, id INT NOT NULL, address VARCHAR(255), email VARCHAR(255), password VARCHAR(255), name VARCHAR(255), city VARCHAR(255), longitude DOUBLE, state VARCHAR(255), source VARCHAR(255), birth_date DATE, zip VARCHAR(255), latitude DOUBLE, CLUSTERING INDEX idx01 (id) ); CREATE TABLE public.products( created_at TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, id INT NOT NULL, ean VARCHAR(255), title VARCHAR(255), category VARCHAR(255) COMPRESSION GDC(2) NOT NULL, vendor VARCHAR(255), price DOUBLE, rating DOUBLE, CLUSTERING INDEX idx01 (category) ); CREATE TABLE public.reviews ( created_at TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, id INT NOT NULL, product_id INT NOT NULL, reviewer VARCHAR(255), rating INT, body VARCHAR(255), CLUSTERING INDEX idx01 (product_id) ); ``` Now, the database tables are created, and you can begin loading data. ## Step 3: Create a Data Pipeline Data pipelines are created using a simple loading configuration that is submitted to the Transformation Nodes to start loading. Each Kafka topic is routed to one or more Ocient tables, and each column is the result of a transformation applied to the source document. First, inspect the data that you load. Each document has a format similar to the following example. ```json JSON theme={null} /* orders */ {"id": 1, "user_id": 1, "product_id": 14, "subtotal": 37.65, "tax": 2.07, "total": 39.72, "discount": null, "created_at": "2019-02-11T21:40:27.892Z", "quantity": 2} {"id": 2, "user_id": 1, "product_id": 123, "subtotal": 110.93, "tax": 6.1, "total": 117.03, "discount": null, "created_at": "2018-05-15T08:04:04.580Z", "quantity": 3} ... /* products */ {"id": 1, "ean": "1018947080336", "title": "Rustic Paper Wallet", "category": "Gizmo", "vendor": "Swaniawski, Casper and Hilll", "price": 29.46, "rating": 4.6, "created_at": "2017-07-19T19:44:56.582Z"} {"id": 2, "ean": "7663515285824", "title": "Small Marble Shoes", "category": "Doohickey", "vendor": "Balistreri-Ankunding", "price": 70.08, "rating": 0, "created_at": "2019-04-11T08:49:35.932Z"} {"id": 3, "ean": "4966277046676", "title": "Synergistic Granite Chair", "category": "Doohickey", "vendor": "Murray, Watsica and Wunsch", "price": 35.39, "rating": 4, "created_at": "2018-09-08T22:03:20.239Z"} ... /* people */ {"id": 1, "address": "9611-9809 West Rosedale Road", "email": "borer-hudson@yahoo.com", "password": "ccca881f-3e4b-4e5c-8336-354103604af6", "name": "Hudson Borer", "city": "Wood River", "longitude": -98.5259864, "state": "NE", "source": "Twitter", "birth_date": "1986-12-12", "zip": "68883", "latitude": 40.71314890000001, "created_at": "2017-10-07T01:34:35.462Z"} {"id": 2, "address": "101 4th Street", "email": "williamson-domenica@yahoo.com", "password": "eafc45bf-cf8e-4c96-ab35-ce44d0021597", "name": "Domenica Williamson", "city": "Searsboro", "longitude": -92.6991321, "state": "IA", "source": "Affiliate", "birth_date": "1967-06-10", "zip": "50242", "latitude": 41.5813224, "created_at": "2018-04-09T12:10:05.167Z"} {"id": 3, "address": "29494 Anderson Drive", "email": "lina.heaney@yahoo.com", "password": "36f67891-34e5-4439-a8a4-2d9246775ff8", "name": "Lina Heaney", "city": "Sandstone", "longitude": -92.8416108, "state": "MN", "source": "Facebook", "birth_date": "1961-12-18", "zip": "55072", "latitude": 46.11973039999999, "created_at": "2017-06-27T06:06:20.625Z"} ... /* reviews */ {"id": 1, "product_id": 1, "reviewer": "christ", "rating": 5, "body": "Ad perspiciatis quis et consectetur. Laboriosam fuga voluptas ut et modi ipsum. Odio et eum numquam eos nisi. Assumenda aut magnam libero maiores nobis vel beatae officia.", "created_at": "2018-05-15T20:25:48.517Z"} {"id": 2, "product_id": 1, "reviewer": "xavier", "rating": 4, "body": "Reprehenderit non error architecto consequatur tempore temporibus. Voluptate ut accusantium quae est. Aut sit quidem nihil maxime dolores molestias. Enim vel optio est fugiat vitae cumque ut. Maiores laborum rerum quidem voluptate rerum.", "created_at": "2019-08-07T13:50:33.401Z"} {"id": 3, "product_id": 1, "reviewer": "cameron.nitzsche", "rating": 5, "body": "In aut numquam labore fuga. Et tempora sit et mollitia aut ullam et repellat. Aliquam sint tenetur culpa eius tenetur. Molestias ipsa est ut quisquam hic necessitatibus. Molestias maiores vero nesciunt.", "created_at": "2018-03-30T00:28:45.192Z"} ... ``` As you can see, this is similar to the target schema, but will require some transformation. Most transformations are identical to functions already in Ocient’s SQL dialect. To route data to your tables, you must create a pipeline.json file that has the following structure: ```json JSON theme={null} { "version": 2, "workers": 4, "pipeline_id": "pipeline-metabase", "source": { "type": "kafka", "kafka": { "bootstrap.servers": "127.0.0.1:9092", "auto.offset.reset": "earliest" } }, "transform": { "topics": { "orders": { "tables": { "metabase.public.orders": { "columns": { "id": "id", "user_id": "user_id", "product_id": "product_id", "subtotal": "subtotal", "tax": "tax", "total": "total", "discount": "discount", "created_at": "to_timestamp(created_at, 'yyyy-MM-dd\\'T\\'HH:mm:ss[.SSS]X')", "quantity": "quantity" } } } }, "people": { "tables": { "metabase.public.people": { "columns": { "id": "id", "address": "address", "email": "email", "password": "password", "name": "name", "city": "city", "longitude": "longitude", "state": "state", "source": "source", "birth_date": "birth_date", "zip": "zip", "latitude": "latitude", "created_at": "to_timestamp(created_at, 'yyyy-MM-dd\\'T\\'HH:mm:ss[.SSS]X')" } } } }, "reviews": { "tables": { "metabase.public.reviews": { "columns": { "id": "id", "product_id": "product_id", "reviewer": "reviewer", "rating": "rating", "body": "body", "created_at": "to_timestamp(created_at, 'yyyy-MM-dd\\'T\\'HH:mm:ss[.SSS]ZZZZZ')" } } } }, "products": { "tables": { "metabase.public.products": { "columns": { "id": "id", "ean": "ean", "title": "title", "category": "category", "vendor": "vendor", "price": "price", "rating": "rating", "created_at": "to_timestamp(created_at, 'yyyy-MM-dd\\'T\\'HH:mm:ss[.SSS]X')" } } } } } } } ``` ## Step 4: Using the Loading and Transformation CLI With a pipeline.json file ready to go, you can test this pipeline. To test, use the LAT CLI. For these examples, you can assume that two LATs are configured and will set them using an environment variable. First, configure the LAT CLI to use the hosts of your Loading and Transformation service. You can add these to every CLI command as a flag, but for simplicity you can also set them as environment variables. From a command line, run the following command replacing the IP addresses with the IP addresses of your LAT processes: ```shell Shell theme={null} export LAT_HOSTS="https://10.0.0.1:8443,https://10.0.0.2:8443" ``` If your LAT is running without TLS configured, replace the port number of your LAT Hosts with 8080 and the protocol with `http://`. Next, check on the status of the LAT: ```shell Shell theme={null} lat_client pipeline status ``` **Example response:** ```bash Bash theme={null} 10.0.0.1:8443: Stopped 10.0.0.2:8443: Stopped ``` Success! This confirms that you can reach the LAT from your CLI. If the status is "Running" it means a pipeline is already executing a pipeline. You are next going to update and start your new pipeline. This example uses secure connections. If you receive an SSL Error when testing, your service cannot be configured to use TLS or you might need to use the `--no-verify` flag if certificate validation fails. ## Step 5: Test the Transformation The CLI supports previewing a transformation with an example document and the pipeline file. This makes it easy to test your transformations. First, save an example document to your file system to use for this test. For this demo, you can download an example file from [https://ocient-docs.s3.amazonaws.com/metabase\_samples/jsonl/orders.jsonl](https://ocient-docs.s3.amazonaws.com/metabase_samples/jsonl/orders.jsonl) and save it to `~/orders.jsonl`. Next, make sure the pipeline.json file that you created is stored at `~/pipeline.json`. Now that both files are available, run the CLI to preview the results. You can pass the preview command the topic name, the pipeline file, and the sample record file. The response contains the transformed data tied to the destination table and a list of any error records. ```shell Shell theme={null} lat_client preview --topic orders --pipeline ~/pipeline.json --records ~/orders.jsonl ``` Example response: ```json JSON theme={null} { "tableRecords": { "metabase.public.orders": [ { "id": 1, "user_id": 1, "product_id": 14, "subtotal": 37.65, "tax": 2.07, "total": 39.72, "discount": null, "created_at": 1549921227892000000, "quantity": 2 }, { "id": 2, "user_id": 1, "product_id": 123, "subtotal": 110.93, "tax": 6.1, "total": 117.03, "discount": null, "created_at": 1526371444580000000, "quantity": 3 }, { "id": 3, "user_id": 1, "product_id": 105, "subtotal": 52.72, "tax": 2.9, "total": 49.2, "discount": 6.42, "created_at": 1575670968544000000, "quantity": 2 } ] }, "recordErrors": [] } ``` You can see that the data is transformed and the columns to which each transformed value will be mapped. If there are issues in the values, these will appear in the `recordErrors` object. You can quickly update your pipeline.json file and preview again. Now, you can inspect different documents to confirm that various states of data cleanliness like missing columns, null values, and special characters are well handled by your transformations. ## Step 6: Configure and Start the Data Pipeline With a tested transformation, the next step is to set up and start the data pipeline. First, configure the pipeline using the `pipeline create` command. This validates and creates the pipeline, but will not take effect until you start the pipeline: ```shell Shell theme={null} lat_client pipeline create --pipeline ~/pipeline.json ``` Example response: ```bash Bash theme={null} 10.0.0.1:8443: Created 10.0.0.2:8443: Created ``` In cases where there is an existing pipeline operating, it is necessary to stop the pipeline and remove the original pipeline before creating and starting the new pipeline. Now that the pipeline has been created on all LAT Nodes, you can start the LAT by running the `pipeline start` commands: ```shell Shell theme={null} lat_client pipeline start ``` Example responses: ```bash Bash theme={null} 10.0.0.1:8443: Running 10.0.0.2:8443: Running ``` ## Step 7: Confirm that Loading is Operating Correctly With your pipeline in place and running, data will immediately begin loading off of the Kafka topics that are configured in the pipeline. If you do not have data in the Kafka topics yet, now would be a good time to start producing data into the topics. ### Producing Test Data into Kafka: For test purposes, [kafkacat](https://github.com/edenhill/kcat) is a helpful utility that makes it easy to product records into a topic. For example, if you have a file of sample data `orders.jsonl` in a JSONL format (newline delimited JSON records), you can run the following command to send those records into your Kafka broker: ```shell Shell theme={null} kafkacat -b :9092 -t -T -P -l orders.jsonl ``` Assuming your broker is running at `10.0.0.3` and you want to send data into the four topics defined in your `pipeline.json` definition, you can run: ```shell Shell theme={null} kafkacat -b 10.0.0.3:9092 -t orders -T -P -l orders.jsonl kafkacat -b 10.0.0.3:9092 -t products -T -P -l products.jsonl kafkacat -b 10.0.0.3:9092 -t people -T -P -l people.jsonl kafkacat -b 10.0.0.3:9092 -t reviews -T -P -l reviews.jsonl ``` Each of these commands will push the entire JSONL file of messages into Kafka with one record per line. As these are produced into Kafka, your running pipeline will begin loading them into Ocient. ### Observing Loading Progress: With data in Kafka, our pipeline will begin loading data immediately and streaming any new data into Ocient. To observe this progress, you can monitor the metrics endpoint of the Loading and Transformation Nodes. This can be done manually from a command line or from a tool like . For this example, you can run a `curl` command against the endpoint and review the result. For details on metrics, see the [LAT Metrics](/lat-metrics) Documentation. **Command:** ```curl CURL theme={null} curl https://127.0.0.1:8443/v2/metrics/lat:type=pipeline ``` If your LAT is running without TLS configured, replace the port number of your LAT Hosts with 8080 and the protocol with `http://`. **Example response:** ```json JSON theme={null} { "request": { "mbean": "lat:type=pipeline", "type": "read" }, "value": { "partitions": [ { "offsets_durable": 1, "pushes_errors": 0, "pushes_attempts": 1, "rows_pushed": 1, "offsets_written": 18759, "records_buffered": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "source_bytes_buffered": 0, "records_errors_transformation": 0, "offsets_processed": 18759, "partition": "table_orders-0", "records_filter_accepted": 1, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "offsets_pushed": 18759, "pushes_unacknowledged": 0, "invalid_state": 0, "bytes_pushed": 88, "records_errors_total": 0, "offsets_buffered": 18759, "complete": 0, "offsets_end": 1, "producer_send_errors": 0 }, { "offsets_durable": 1, "pushes_errors": 0, "pushes_attempts": 1, "rows_pushed": 1, "offsets_written": 2499, "records_buffered": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "source_bytes_buffered": 0, "records_errors_transformation": 0, "offsets_processed": 2499, "partition": "table_people-0", "records_filter_accepted": 1, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "offsets_pushed": 2499, "pushes_unacknowledged": 0, "invalid_state": 0, "bytes_pushed": 223, "records_errors_total": 0, "offsets_buffered": 2499, "complete": 0, "offsets_end": 1, "producer_send_errors": 0 }, { "offsets_durable": 1, "pushes_errors": 0, "pushes_attempts": 1, "rows_pushed": 1, "offsets_written": 199, "records_buffered": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "source_bytes_buffered": 0, "records_errors_transformation": 0, "offsets_processed": 199, "partition": "table_products-0", "records_filter_accepted": 1, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "offsets_pushed": 199, "pushes_unacknowledged": 0, "invalid_state": 0, "bytes_pushed": 145, "records_errors_total": 0, "offsets_buffered": 199, "complete": 0, "offsets_end": 1, "producer_send_errors": 0 }, { "offsets_durable": 1, "pushes_errors": 0, "pushes_attempts": 1, "rows_pushed": 1, "offsets_written": 1111, "records_buffered": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "source_bytes_buffered": 0, "records_errors_transformation": 0, "offsets_processed": 1111, "partition": "table_reviews-0", "records_filter_accepted": 1, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "offsets_pushed": 1111, "pushes_unacknowledged": 0, "invalid_state": 0, "bytes_pushed": 184, "records_errors_total": 0, "offsets_buffered": 1111, "complete": 0, "offsets_end": 1, "producer_send_errors": 0 } ], "paused": 1, "bytes_buffered": 0, "workers": 20 }, "timestamp": 1626970368, "status": 200 } ``` ### Check Row Counts in Tables: To confirm that you are seeing results in the target tables, you can also run some simple queries to check row counts. Depending on the streamloader role settings, the time for records to become queryable can vary from a few seconds to minutes: **Example Queries:** ```sql SQL theme={null} Ocient> SELECT count(*) FROM public.orders; count(*) ------------------------ 18760 ``` ```sql SQL theme={null} Ocient> SELECT count() FROM public.people; count() ----------------------- 2500 ``` ```sql SQL theme={null} Ocient> SELECT count(*) FROM public.products; count(*) -------------------- 200 ``` ```sql SQL theme={null} Ocient> SELECT count(*) FROM public.reviews; count(*) -------------------- 1112 ``` Success! Now you can explore the data in these four tables with any Ocient SQL queries. If more data is pushed into these topics, your pipeline is still running and will automatically load all new data. ### Check Errors In this example, all rows load successfully. However, a successful load does not always happen, and you can inspect errors using the LAT Client. Whenever the LAT process fails to parse a file correctly or fails to transform or load a record, the LAT process records an error. The LAT Client includes the `lat_client pipeline errors` command that reports the latest errors on the pipeline. A full error log is available on the Loader Nodes. These logs report all bad records and the reason that the load fails. When you load a pipeline from Kafka, the load might route errors to an error topic on the Kafka broker instead of the logs. The LAT Client does not contain the errors sent to the error topic. You can inspect these errors with Kafka utilities instead. This LAT Client command displays a maximum of 100 error messages. ```bash theme={null} theme={null} lat_client pipeline errors --max-errors 100 --only-error-messages |--------------------------------------------------| | exception_message | |--------------------------------------------------| | Column name: time1. Message: Failed to evaluate | | expression. Cause: | | java.time.format.DateTimeParseException | |--------------------------------------------------| | Column name: time1. Message: Failed to evaluate | | expression. Cause: | | java.time.format.DateTimeParseException | |--------------------------------------------------| ``` The errors indicate that there is an issue parsing the `time1` column. Options exist on the `pipeline errors` command to return JSON and to restrict the response to specific components of the error detail that includes a reference to the source location of this record. The following command returns JSON that is delimited with newline characters. You can pass the JSON output to `jq` or a file. The JSON includes the source topic or file group, the filename where the error occurred, the offset that indicates the line number or Kafka offset, and the exception message that aids in troubleshooting and identifying the incorrect record in the source data. You can use the `log_original_message` pipeline setting to provide direct access to the parsed source record for errors when appropriate. ```bash theme={null} theme={null} lat_client pipeline errors --max-errors 100 --json {"time": "2022-05-17T16:53:50.387386+00:00", "topic": "calcs", "partition": 0, "state": "TRANSFORMATION_ERROR", "exception_message": "Column name: time1. Message: Failed to evaluate expression. Cause: java.time.format.DateTimeParseException: Cannot parse time \"19:36:22\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\njava.time.format.DateTimeParseException: Cannot parse time \"19:36:22\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\nCannot parse time \"19:36:22\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"", "offset": 0, "record": null, "metadata": {"size": "3321", "filename": "calcs/csv/calcs_01.csv"}} {"time": "2022-05-17T16:53:50.404684+00:00", "topic": "calcs", "partition": 0, "state": "TRANSFORMATION_ERROR", "exception_message": "Column name: time1. Message: Failed to evaluate expression. Cause: java.time.format.DateTimeParseException: Cannot parse time \"02:05:25\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\njava.time.format.DateTimeParseException: Cannot parse time \"02:05:25\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\nCannot parse time \"02:05:25\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"", "offset": 1, "record": null, "metadata": {"size": "3321", "filename": "calcs/csv/calcs_01.csv"}} ``` ## Related Links [LAT Overview](/lat-overview) [LAT Data Types in Loading](/lat-data-types-in-loading) [LAT Advanced Topics](/lat-advanced-topics) # LAT Load JSON Data from S3 Source: https://docs.ocient.com/lat-load-json-data-from-s3 Integrate JSON data from Amazon S3 into Ocient for advanced analytics, designed for high-throughput and efficient processing of large data sets. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). A common setup for batch loading files into Ocient is to load from a bucket on S3 with time partitioned data. In many instances, a batch load is performed on a recurring basis to load new files. The LAT transforms each document into rows in one or more different tables. Ocient’s Loading and Transformation capabilities use a simple SQL-like syntax for transforming data. This tutorial will guide users through a simple example load using a small set of data in JSONL (newline delimited JSON) format. The data in this example is created from a test set for the Business Intelligence tool. ## Prerequisites This tutorial assumes that: 1. The Ocient System has network access to S3 from the Loader Nodes. 2. An Ocient System is installed and configured with an active storage cluster (See the [Ocient Application Configuration](/ocient-application-configuration) guide). 3. Loading and Transformation is installed on the Loader Nodes. 4. A default "sink" for the Ocient Loader Nodes is configured on the system. 5. The [LAT Client Command Line Interface](/lat-client-command-line-interface) is installed. ## Step 1: Create a New Database To begin, you are going to load two example tables in a database. First, connect to a SQL Node using the [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). Then run the following DDL command: ```sql SQL theme={null} CREATE DATABASE metabase; ``` ## Step 2: Create Tables To create tables in the new database, first connect to that database (e.g., `connect to jdbc:ocient://sql-node:4050/metabase`), then run the following DDL commands: ```sql SQL theme={null} CREATE TABLE public.orders ( created_at TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, id INT NOT NULL, user_id INT NOT NULL, product_id INT NOT NULL, subtotal DOUBLE, tax DOUBLE, total DOUBLE, discount DOUBLE, quantity INT, CLUSTERING INDEX idx01 (user_id, product_id) ); CREATE TABLE public.products( created_at TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, id INT NOT NULL, ean VARCHAR(255), title VARCHAR(255), category VARCHAR(255) COMPRESSION GDC(2) NOT NULL, vendor VARCHAR(255), price DOUBLE, rating DOUBLE, CLUSTERING INDEX idx01 (category) ); ``` Now, the database tables are created and you can begin loading data. ## Step 3: Create a Data Pipeline Data pipelines are created using a simple loading configuration that is submitted to the Transformation Nodes to start loading. File Groups designate a batch of files to load. Each File Group is routed to one or more Ocient tables, and each column is the result of a transformation applied to the source document. First, let’s inspect the data you plan to load. Each document has a format similar to the following example: ```json JSON theme={null} /* orders */ {"id": 1, "user_id": 1, "product_id": 14, "subtotal": 37.65, "tax": 2.07, "total": 39.72, "discount": null, "created_at": "2019-02-11T21:40:27.892Z", "quantity": 2} {"id": 2, "user_id": 1, "product_id": 123, "subtotal": 110.93, "tax": 6.1, "total": 117.03, "discount": null, "created_at": "2018-05-15T08:04:04.580Z", "quantity": 3} ... /* products */ {"id": 1, "ean": "1018947080336", "title": "Rustic Paper Wallet", "category": "Gizmo", "vendor": "Swaniawski, Casper and Hilll", "price": 29.46, "rating": 4.6, "created_at": "2017-07-19T19:44:56.582Z"} {"id": 2, "ean": "7663515285824", "title": "Small Marble Shoes", "category": "Doohickey", "vendor": "Balistreri-Ankunding", "price": 70.08, "rating": 0, "created_at": "2019-04-11T08:49:35.932Z"} {"id": 3, "ean": "4966277046676", "title": "Synergistic Granite Chair", "category": "Doohickey", "vendor": "Murray, Watsica and Wunsch", "price": 35.39, "rating": 4, "created_at": "2018-09-08T22:03:20.239Z"} ... ``` As you can see, this is similar to your target schema, but will require some transformation. Most transformations are identical to functions already in Ocient’s SQL dialect. To route data to your tables, you need to create a pipeline.json file that has the following structure: ```json JSON theme={null} { "version": 2, "workers": 4, "source": { "type": "s3", "endpoint": "https://s3.us-east-1.amazonaws.com", "bucket": "ocient-docs", "compression": "none", "file_groups": { "orders": { "prefix": "metabase_samples/jsonl", "file_matcher_syntax": "glob", "file_matcher_pattern": "**orders*.jsonl", "sort_type": "lexicographic" }, "products": { "prefix": "metabase_samples/jsonl", "file_matcher_syntax": "glob", "file_matcher_pattern": "**products*.jsonl", "sort_type": "lexicographic" } } }, "transform": { "file_groups": { "orders": { "tables": { "metabase.public.orders": { "columns": { "id": "id", "user_id": "user_id", "product_id": "product_id", "subtotal": "subtotal", "tax": "tax", "total": "total", "discount": "discount", "created_at": "to_timestamp(created_at, 'yyyy-MM-dd\\'T\\'HH:mm:ss[.SSS]X')", "quantity": "quantity" } } } }, "products": { "tables": { "metabase.public.products": { "columns": { "id": "id", "ean": "ean", "title": "title", "category": "category", "vendor": "vendor", "price": "price", "rating": "rating", "created_at": "to_timestamp(created_at, 'yyyy-MM-dd\\'T\\'HH:mm:ss[.SSS]X')" } } } } } } } ``` The most interesting part of this pipeline.json file is the way it defines the file groups. Note that each sets the S3 endpoint, a bucket, a prefix used for filtering the considered files, and then a file matcher. In this case you only have a single file, but if there were many files matching the pattern `**orders*.jsonl` then they would all be part of the file group. The final parameter that you supplied is the sort type for the file load. This informs the LAT how you would like data to be ordered when loading. The ideal sort is in time order according to the defined . This makes more efficient segments and is much faster to load. In this case, you used the lexicographic sort which orders according to the characters in the filename. Other sort types are available to use file modified time or to extract the timestamp for sorting from the file path or filename. ## Step 4: Using the Loading and Transformation CLI With a pipeline.json file ready to go, you can test this pipeline. To test, use the LAT CLI. For these examples, assume that two LATs are configured and set using an environment variable. First configure the LAT CLI to use the hosts of the Ocient Loading and Transformation service. You can add these to every CLI command as a flag, but for simplicity you can also set them as environment variables. From a command line, run the following command replacing the IP addresses with the IP addresses of your LAT processes: ```shell Shell theme={null} export LAT_HOSTS="https://10.0.0.1:8443,https://10.0.0.2:8443" ``` If your LAT is running without TLS configured, replace the port number of your LAT Hosts with 8080 and the protocol with `http://`. Next, check on the status of the LAT: ```shell Shell theme={null} lat_client pipeline status ``` **Example response:** ```bash Bash theme={null} 10.0.0.1:8443: Stopped 10.0.0.2:8443: Stopped ``` This confirms that you can reach the LAT from your CLI. If the status is "Running" it means a pipeline is already executing a pipeline. Next, you are going to update and start your new pipeline. This example uses secure connections. If you receive an SSL Error when testing, your service might not be configured to use TLS or you might need to use the `--no-verify` flag if the certificate validation fails. ## Step 5: Test the Transformation The CLI supports previewing a transformation with an example document and the pipeline file. This makes it easy to test your transformations. First, save an example document to your file system to use for this test. For this demo, you can download an example file from [https://ocient-docs.s3.amazonaws.com/metabase\_samples/jsonl/orders.jsonl](https://ocient-docs.s3.amazonaws.com/metabase_samples/jsonl/orders.jsonl) and save it to `~/orders.jsonl`. Next, make sure the pipeline.json file that you created is stored at `~/pipeline.json`. Now that both files are available, you can run the CLI to preview the results. Pass the preview command the topic name, the pipeline file, and the sample record file. The response contains the transformed data tied to the destination table and a list of any error records. Similar to how you can preview records on a topic for file loads, you can supply any one of the topics you created as file groups to preview the transformations. ```shell Shell theme={null} lat_client preview --topic orders --pipeline ~/pipeline.json --records ~/orders.jsonl ``` **Example response:** ```json JSON theme={null} { "tableRecords": { "metabase.public.orders": [ { "id": 1, "user_id": 1, "product_id": 14, "subtotal": 37.65, "tax": 2.07, "total": 39.72, "discount": null, "created_at": 1549921227892000000, "quantity": 2 }, { "id": 2, "user_id": 1, "product_id": 123, "subtotal": 110.93, "tax": 6.1, "total": 117.03, "discount": null, "created_at": 1526371444580000000, "quantity": 3 }, { "id": 3, "user_id": 1, "product_id": 105, "subtotal": 52.72, "tax": 2.9, "total": 49.2, "discount": 6.42, "created_at": 1575670968544000000, "quantity": 2 } ] }, "recordErrors": [] } ``` You can see that the data is transformed and the columns to which each transformed value will be mapped. If there are issues in the values, these will appear in the `recordErrors` object. You can quickly update your pipeline.json file and preview again. Now, you can inspect different documents to confirm that various states of data cleanliness like missing columns, null values, and special characters are well handled by your transformations. ## Step 6: Configure and Start the Data Pipeline With a tested transformation, the next step is to setup and start the data pipeline. First, you must configure the pipeline using the `pipeline create` command. This validates and creates the pipeline, but will not take effect until you start the pipeline: ```shell Shell theme={null} lat_client pipeline create --pipeline ~/pipeline.json ``` **Example response:** ```bash Bash theme={null} 10.0.0.1:8443: Created 10.0.0.2:8443: Created ``` In cases where there is an existing pipeline operating, it is necessary to stop the pipeline and remove the original pipeline before creating and starting the new pipeline. Now that the pipeline has been created on all LAT Nodes, you can start the LAT by running the `pipeline start` commands: ```shell Shell theme={null} lat_client pipeline start ``` Example responses: ```bash Bash theme={null} 10.0.0.1:8443: Running 10.0.0.2:8443: Running ``` ## Step 7: Confirm that Loading is Operating Correctly With your pipeline in place and running, data will immediately begin loading from the S3 file groups you defined. If there were many files per file group, the LAT would first sort the files, then partition them for the fastest loading based on the sorting criteria you provided. ### Observing Loading Progress With the pipeline running, data immediately begins to load into Ocient. To observe this progress, you can use the `pipeline status` command from the LAT Client or monitor the LAT metrics endpoint of the Loader Nodes. You can check the status with this command by using the `--list-files` flag to include a summary of the files included in the load. ```shell Shell theme={null} lat_client pipeline status --list-files ``` **Example responses:** ```bash Bash theme={null} 10.0.0.1:8443: Running 10.0.0.2:8443: Running Pipeline Files Processed: 0 Pipeline Error Count: 0 Pipeline Files Remaining: 2 orders Files Processed: 0 Error Count: 0 Files Remaining: 1 products Files Processed: 0 Error Count: 0 Files Remaining: 1 orders Status Filename processing metabase_samples/jsonl/orders.jsonl In Process Files: processing metabase_samples/jsonl/orders.jsonl products Status Filename processing metabase_samples/jsonl/products.jsonl In Process Files: processing metabase_samples/jsonl/products.jsonl ``` **Command:** ```curl CURL theme={null} curl https://127.0.0.1:8443/v2/metrics/lat:type=pipeline ``` If your LAT is running without TLS configured, replace the port number of your LAT Hosts with 8080 and the protocol with `http://`. **Example response:** ```json JSON theme={null} { "request": { "mbean": "lat:type=pipeline", "type": "read" }, "value": { "partitions": [ { "offsets_durable": 1, "pushes_errors": 0, "pushes_attempts": 1, "rows_pushed": 1, "offsets_written": 18759, "records_buffered": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "source_bytes_buffered": 0, "records_errors_transformation": 0, "offsets_processed": 18759, "partition": "table_orders-0", "records_filter_accepted": 1, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "offsets_pushed": 18759, "pushes_unacknowledged": 0, "invalid_state": 0, "bytes_pushed": 88, "records_errors_total": 0, "offsets_buffered": 18759, "complete": 0, "offsets_end": 1, "producer_send_errors": 0 }, { "offsets_durable": 1, "pushes_errors": 0, "pushes_attempts": 1, "rows_pushed": 1, "offsets_written": 199, "records_buffered": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "source_bytes_buffered": 0, "records_errors_transformation": 0, "offsets_processed": 199, "partition": "table_products-0", "records_filter_accepted": 1, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "offsets_pushed": 199, "pushes_unacknowledged": 0, "invalid_state": 0, "bytes_pushed": 145, "records_errors_total": 0, "offsets_buffered": 199, "complete": 0, "offsets_end": 1, "producer_send_errors": 0 } ], "paused": 1, "bytes_buffered": 0, "workers": 20 }, "timestamp": 1626970368, "status": 200 } ``` ### Check Row Counts in Tables To confirm that you are seeing results in the target tables, you can also run some simple queries to check row counts. Depending on the streamloader role settings, the time for records to become queryable can vary from a few seconds to minutes: **Example Queries:** ```sql SQL theme={null} Ocient> SELECT count(*) FROM public.orders; COUNT(*) ---------------------- 18760 ``` ```sql SQL theme={null} Ocient> SELECT count() FROM public.products; COUNT() -------------------- 200 ``` Now you can explore the data in these four tables with any Ocient SQL queries. ### Check Errors In this example, all rows load successfully. However, a successful load does not always happen, and you can inspect errors using the LAT Client. Whenever the LAT process fails to parse a file correctly or fails to transform or load a record, the LAT process records an error. The LAT Client includes the `lat_client pipeline errors` command that reports the latest errors on the pipeline. A full error log is available on the Loader Nodes. These logs report all bad records and the reason that the load fails. When you load a pipeline from Kafka, the load might route errors to an error topic on the Kafka broker instead of the logs. The LAT Client does not contain the errors sent to the error topic. You can inspect these errors with Kafka utilities instead. This LAT Client command displays a maximum of 100 error messages. ```bash theme={null} theme={null} lat_client pipeline errors --max-errors 100 --only-error-messages |--------------------------------------------------| | exception_message | |--------------------------------------------------| | Column name: time1. Message: Failed to evaluate | | expression. Cause: | | java.time.format.DateTimeParseException | |--------------------------------------------------| | Column name: time1. Message: Failed to evaluate | | expression. Cause: | | java.time.format.DateTimeParseException | |--------------------------------------------------| ``` The errors indicate that there is an issue parsing the `time1` column. Options exist on the `pipeline errors` command to return JSON and to restrict the response to specific components of the error detail that includes a reference to the source location of this record. The following command returns JSON that is delimited with newline characters. You can pass the JSON output to `jq` or a file. The JSON includes the source topic or file group, the filename where the error occurred, the offset that indicates the line number or Kafka offset, and the exception message that aids in troubleshooting and identifying the incorrect record in the source data. You can use the `log_original_message` pipeline setting to provide direct access to the parsed source record for errors when appropriate. ```bash theme={null} theme={null} lat_client pipeline errors --max-errors 100 --json {"time": "2022-05-17T16:53:50.387386+00:00", "topic": "calcs", "partition": 0, "state": "TRANSFORMATION_ERROR", "exception_message": "Column name: time1. Message: Failed to evaluate expression. Cause: java.time.format.DateTimeParseException: Cannot parse time \"19:36:22\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\njava.time.format.DateTimeParseException: Cannot parse time \"19:36:22\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\nCannot parse time \"19:36:22\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"", "offset": 0, "record": null, "metadata": {"size": "3321", "filename": "calcs/csv/calcs_01.csv"}} {"time": "2022-05-17T16:53:50.404684+00:00", "topic": "calcs", "partition": 0, "state": "TRANSFORMATION_ERROR", "exception_message": "Column name: time1. Message: Failed to evaluate expression. Cause: java.time.format.DateTimeParseException: Cannot parse time \"02:05:25\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\njava.time.format.DateTimeParseException: Cannot parse time \"02:05:25\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"\nCannot parse time \"02:05:25\" with format string \"Value(HourOfDay,2)Offset(+HHmm,'Z')':'Value(MinuteOfHour,2)':'Value(SecondOfMinute,2)\"", "offset": 1, "record": null, "metadata": {"size": "3321", "filename": "calcs/csv/calcs_01.csv"}} ``` ## Related Links [LAT Overview](/lat-overview) [LAT Data Types in Loading](/lat-data-types-in-loading) [LAT Advanced Topics](/lat-advanced-topics) # LAT Log4J Configuration Source: https://docs.ocient.com/lat-log4j-configuration Configure Log4j logging for the Ocient Loading and Transformation (LAT) system to control log levels, appenders, and rotation for troubleshooting and audit. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). The LAT uses for [logging](https://logging.apache.org/log4j/2.x/). Users can configure their logging setup however they would like. The LAT packages come with example `log4j2.xml` files that can be used directly, or modified by the user. This XML code in the `log4j2.xml` file shows an example configuration. ```xml XML theme={null} ``` Relevant Loggers: * `org.apache.kafka`: logger. * `com.ocient.lat.sink.ocient.Binders.Binder`: LAT binder logger. Can be verbose, recommended to keep at `info` level. * `org.eclipse.jetty.server.HttpChannel` and `org.eclipse.jetty.server.RequestLog`: Jetty logs for the HTTP endpoint. * `com.ocient.lat.Worker`: Main worker logger containing important pipeline logs. Valid log levels are (in order of least to most verbose): * `error` * `warn` * `info` * `debug` * `trace` Log4j can be configured to log different packages at different levels as shown in the example. Here, the example logs the `com.ocient.streaming.client` package at the trace level, but the remainder of the code is logged at info level. ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Metrics Source: https://docs.ocient.com/lat-metrics Track Ocient LAT pipeline metrics such as record counts, throughput, errors, and latency to monitor data loading performance and detect issues quickly. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). ## Configuration LAT metrics are exposed using the LAT REST API. ## Endpoints LAT exposes two metrics endpoints; a ***partitions*** endpoint, and a ***pipeline*** endpoint. ### Partition Metrics The partitions endpoint exposes individual metrics for every partition that is participating in the current LAT instance. For loads, partitions correspond to Kafka’s partitions. For File Source based loads, partitions correspond to the independent file sets created within each File Group for parallel processing. The partitions metrics endpoint can be accessed by issuing a curl request as follows: ```curl CURL theme={null} curl http://127.0.0.1:8080/v2/metrics/read/lat:type=partitions ``` A response for this request will look as follows, with one entry per partition in the partitions array: ```json JSON theme={null} { "request": { "mbean": "lat:type=partitions", "type": "read" }, "value": { "partitions": [ { "offsets_durable": 0, "pushes_errors": 0, "pushes_attempts": 0, "rows_pushed": 0, "offsets_written": 0, "records_buffered": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "records_errors_transformation": 0, "offsets_processed": 0, "lag": 0, "partition": "topic-0", "records_filter_accepted": 0, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "offsets_pushed": 0, "pushes_unacknowledged": 0, "invalid_state": 0, "bytes_pushed": 0, "errors_partition": 0, "records_errors_total": 0, "offsets_buffered": 0, "complete": 0, "offsets_end": 0, "producer_send_errors": 0 } ] }, "timestamp": 1642497992, "status": 200 } ``` ### Partition Metrics Definitions Metrics definitions are as follows: | **Metric** | **Description** | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `partition` | Topic partition pair (e.g., `mytopic-0`) | | `records_errors_deserialization` | Total number of processed records which failed to deserialize | | `records_errors_transformation` | Total number of processed records which failed to transform | | `records_errors_column` | Total number of processed records which failed to bind transformed values to columns | | `records_errors_row` | Total number of processed records which failed during builder row advancement/other builder errors | | `records_errors_generic` | Total number of processed records which failed for any other reason | | `records_errors_total` | Sum of all record errors for this topic | | `records_buffered` | Total number of processed records which have not yet been pushed. Should always be modulo configured buffer size | | `records_filter_rejected` | Number of records rejected by the topic filter. | | `records_filter_accepted` | Number of records accepted by the topic filter. | | `offsets_processed` | For Kafka loading, records are processed if their offsets are strictly less than the value of `offsets_processed`.

For file loading, this represents the index of the most recently processed file.

If there is an error during the processing of a record/file, this metric is still updated according to that record/file’s offset. When a file load is complete, this offset becomes equal to offsets\_end. Can decrease due to reprocessing. | | `offsets_written` | For Kafka loading, records are written if their offsets are strictly less than the value of offsets\_written.

For file loading, this represents the index of the most recently written file.

This metric is only updated when the record/file is processed and the write is successful. Note that a successful write does not mean the record/file is durable. Can decrease due to reprocessing. | | `offsets_buffered` | For Kafka loading, this represents the most recently processed record which was processed to completion and is waiting to be pushed into the .

For file loading, this represents the index of the most recently processed file which was processed to completion and is waiting to be pushed into the Ocient data warehouse. | | `offsets_pushed` | For Kafka loading, this represents the highest offset of the batch of rows most recently pushed to the Ocient data warehouse.

For file loading, this represents the highest file index of the batch of rows most recently pushed to the Ocient data warehouse.

Can decrease due to reprocessing. | | `offsets_end` | For Kafka loading, this represents the end offset of the partition.

For file loading, this represents the file count of the partition.
Should never decrease. | | `offsets_durable` | For Kafka loading, records are made durable if their offsets are strictly less than the value of offsets\_durable.

For file loading, this represents the most durable file index.
Should never decrease. | | `producer_send_attempts` | Number of Kafka error topic producer send attempts | | `producer_send_errors` | Number of Kafka error topic producer send errors | | `bytes_pushed` | Number of bytes pushed into the Ocient data warehouse | | `rows_pushed` | Number of rows pushed into the Ocient data warehouse | | `pushes_attempts` | Number of attempts to push record batches into the Ocient data warehouse for this partition | | `pushes_errors` | Number of attempts to push record batches into the Ocient data warehouse which resulted in error | | `pushes_unacknowledged` | Number of attempts to push record batches into the Ocient data warehouse for which no response has yet been received | | `invalid_state` | Number of times a code path was reached in LAT which is erroneous | | `complete` | Whether or not the partition has any records left to process at the moment; this status can change often in a Kafka load but will likely not change from complete to incomplete in a file load | | `errors_partition` | The number of times the LAT failed to fetch records for a particular partition | | `lag` | Calculated as offsets\_end - offsets\_durable; In a Kafka load, lag represents the number of unprocessed records. In a file load, lag represents the number of unprocessed files.

Lag becomes zero when the complete status is true. | ### Pipeline Metrics The pipeline endpoint exposes aggregate metrics for the LAT instance. There are three categories of pipeline metrics: | **Pipeline Metric Category** | **Description** | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `"pipeline" aggregate` | Monotonically-increasing metrics that act as lifetime counters for the running pipeline. They are independent of the current set of partitions. For instance, `rows_pushed` is the total number of rows ever pushed for this pipeline. | | `"partitions" aggregate` | Metrics that depend on the current set of active partitions for the running pipeline. For instance, `total_lag` is the summed lag across all currently active partitions for the pipeline. | | `pipeline-specific` | Pipeline only metrics that are not aggregates. | The Pipeline Metrics endpoint can be accessed by issuing a curl request as follows: ```shell Shell theme={null} curl http://127.0.0.1:8080/v2/metrics/read/lat:type=pipeline ``` A response for this request will look as follows: ```json JSON theme={null} { "request": { "mbean": "lat:type=pipeline", "type": "read" }, "value": { "pipeline": { "pushes_errors": 0, "pushes_attempts": 0, "rows_pushed": 0, "records_errors_column": 0, "records_errors_deserialization": 0, "records_errors_transformation": 0, "bytes_pushed": 0, "errors_partition": 0, "records_filter_accepted": 0, "records_errors_total": 0, "records_errors_row": 0, "records_filter_rejected": 0, "records_errors_generic": 0, "producer_send_attempts": 0, "producer_send_errors": 0 }, "partitions": { "max_lag": 0, "total_offsets_buffered": 0, "avg_lag": 0.0, "total_pushes_unacknowledged": 0, "min_lag": 0, "total_lag": 0, "total_invalid_state": 0, "total_offsets_pushed": 0, "total_records_buffered": 0, "total_complete": 0, "total_offsets_processed": 0 }, "paused": 0, "bytes_buffered": 0, "complete": 0, "workers": 32, "lat_version": "3.0.0" }, "timestamp": 1642497956, "status": 200 } ``` Individual metrics such as `lag`, `offsets_buffered`, `pushes_errors`, etc. are defined within the [Partition Metrics Definitions](#partition-metrics-definitions) section. However, metrics specific *only* to the pipeline endpoint are defined as follows: | **Metric** | **Description** | | ---------------- | ----------------------------------------------------------------------- | | `paused` | 0 if processing is active across all workers, 1 otherwise. | | `bytes_buffered` | Global allocated memory in bytes. | | `complete` | 1 if the pipeline has finished the entire loading process, 0 otherwise. | | `workers` | The number of active workers. | | `lat_version` | The version of the running LAT server. | ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) [System Information REST Endpoints](/system-information-rest-endpoints) # LAT Overview Source: https://docs.ocient.com/lat-overview Overview of the Ocient Loading and Transformation (LAT) system for building pipelines that ingest, transform, and load data into the data warehouse. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). The Ocient Loading and Transformation (LAT) is a service used for transforming and streaming data into Ocient. The LAT transforms records from a configured Source using the [JMESPath](https://jmespath.org/) transformation language and streams records into Ocient tables. ## LAT Benefits * Exactly-once processing from a Source * Selecting data and arrays from nested JSON data * Transforming records and routing into one or more tables * Scalar data transformation (e.g., concatenate strings, uppercasing, rounding) * Dynamic schema change handling ## LAT Data Flow The LAT consumes JSON records, transforms them, and loads them into one or more Ocient tables. This process is referred to as a "pipeline", with a single pipeline running per LAT instance. The pipeline is described by a JSON-based configuration that instructs the LAT how to consume records from a Source, how to transform the records, and which Ocient table to route the records to. As records are consumed, the transformations described in the configuration are applied to each record before forwarding the resulting records to Ocient. Each pipeline configuration can map a record to 1 or more Ocient Tables, each table with its own transformation configuration. LAT can optionally be configured to write records which were unable to be processed to an error topic. Multiple LAT instances can be used in tandem to further parallelize a pipeline. The LAT makes use of the Kafka Consumer library to ensure that each Kafka Partition is only assigned to a single instance. To use multiple instances, configure each LAT instance with the same pipeline configuration. The LAT instances will automatically coordinate and distribute the load across the instances. The LAT is controlled by an HTTP endpoint and a command line interface (CLI) is supplied to perform common actions such as starting, stopping, or updating the LAT pipelines. ## Related Links [Connect to Ocient](/connect-to-ocient) [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) ## Related Links [At the Whiteboard with Ocient: Loading and Transformation](https://www.youtube.com/watch?v=D3exgC20REc) # LAT Packaging and Installation Source: https://docs.ocient.com/lat-packaging-and-installation Package and install the Ocient Loading and Transformation (LAT) client and service, including standalone, distributed, and disconnected deployment options. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). The Ocient LAT supports multiple types of packages for deployment and installation: * `.deb` package * `.rpm` package * Raw `.tar.gz` tar folder * Direct .jar * container Contact Ocient Support for the latest package for a deployment method. ## Prerequisites * 17 or greater * Can either be system-level Java, or installed somewhere accessible by the user. The LAT must run on Ocient Loader Nodes where you installed the Ocient RPM to provide access to numerical libraries, ensuring the correct loading of geospatial data types. If this condition is not met, some geospatial objects might load inaccurately. ## Debian/CentOS Packages **1. Install/Upgrade** Debian: `sudo dpkg -i lat_$VERSION_all.deb` CentOS: `sudo rpm -i lat-$VERSION-1.noarch.rpm` **2. Setup Java** If the default system Java is version 17 or greater, no action is required. If not, edit the service file to contain a `JAVA_HOME` environment variable pointing to a Java version 17 or greater installation. 1. Run: `systemctl edit lat` 2. Insert the following ```shell Shell theme={null} [Service] Environment=JAVA_HOME="PATH_TO_JAVA_HOME" ``` **3. Configure** Modify the [LAT Source Configuration](/lat-source-configuration) and [LAT Log4J Configuration](/lat-log4j-configuration) (see [Package Contents](#package-contents) for file locations). It is recommended to use the `.conf` file for most LAT configuration. However, `LAT_MEMORY` and `LAT_MEMORY_DIRECT` can only be set by using environment variable. For these use the same process as setting `JAVA_HOME` as detailed in step 2. **4. Start/Stop** Start: `systemctl start lat` Stop: `systemctl stop lat` ### Package Contents * Jar: `/opt/lat/lat-$VERSION.jar` * Service File: `/usr/lib/systemd/system/lat.service` * Configuration: `/etc/lat/` * Server Configuration: `lat.conf` * Logging Configuration: `log4j2.xml` * Logs: `/var/log/lat/lat.log` * LAT data directory: `/opt/lat/.lat-data` ## Tar Folder **1. Extract file in your preferred location** Run: `tar -xzf lat-$VERSION.tar.gz` **2. Setup Java** If the default system Java is version 17 or greater, no action is required. If not, set the `$JAVA_HOME` environment variable to a Java installation with version 17 or greater: `export JAVA_HOME="PATH_TO_JAVA_HOME"` **3. Configure** Modify the [LAT Source Configuration](/lat-source-configuration) and [LAT Log4J Configuration](/lat-log4j-configuration) (see [Folder Contents](#folder-contents) for file locations). It is recommended to use the `.conf` file for most LAT configuration. However, `LAT_MEMORY` and `LAT_MEMORY_DIRECT` can only be set using environment variable. For these use the same process as setting `JAVA_HOME` as detailed in step 2. **4. Start/Stop** Start: `./lat.sh [path]`, where `path` is an optional parameter for the path to a server configuration file (e.g., `lat.conf`). If `path` is not present, it will default to `config/lat.conf`. Stop: Stop the running `lat.sh` script with `ctrl-c`. ### Folder Contents * Jar: `lat-$VERSION.jar` * Configuration: `config/` * Server Configuration: `lat.conf` * Logging Configuration: `log4j2.xml` * Start Script: `lat.sh` * LAT data directory: `.lat-data` ## Jar Direct The LAT is a Java jar that can be deployed on any machine running Java 17+. To run the LAT use the following command line statement: ```shell Shell theme={null} java [options] -Dlog4j.configurationFile= -jar [lat-config-path] ``` * `log4j-config-path`: REQUIRED : path to the log4j configuration file * `lat-jar-path`: REQUIRED : path to the LAT jar * `lat-config-path`: OPTIONAL : path to a LAT service configuration file ### Recommended Java Command Line Options See the [Java command line documentation](https://docs.oracle.com/en/java/javase/17/docs/specs/man/java.html#overview-of-java-options) for options. ### Max Direct Memory Size | Option: | -XX:MaxDirectMemorySize | | ------------ | --------------------------- | | Recommended: | 32G | | Example: | -XX:MaxDirectMemorySize=32G | ### Minimum Heap Size | Option: | -Xms | | ------------ | -------- | | Recommended: | 32G | | Example: | -Xms=32G | ### Maximum Heap Size | Option: | -Xmx | | ------------ | -------- | | Recommended: | 32G | | Example: | -Xmx=32G | ### Example Java Command Line ```shell Shell theme={null} java \ -Dlog4j.configurationFile=config/log4j2.xml \ -XX:MaxDirectMemorySize=32G \ -Xms32G \ -Xmx32G \ -jar lat-[VERSION NUMBER].jar \ config/lat.conf ``` ## Docker Container Ocient provides a Docker container for the LAT for testing workloads. The container is not recommended for production workloads and should be used for testing only. ### Configuration The only mode of configuring the LAT within a Docker container is by using the service configuration environment variables. A `lat.conf` file is not supported. See [LAT Service Configuration](/lat-service-configuration) for the list of supported environment variables. It is not recommended to override the following environment variables. * `LAT_DATA_PATH` * `LAT_API_PORT` ### Logging Within the Docker container the LAT logs `stdout`. The user can use the `docker logs ...` command to view the LAT logs. The following environment variables are available to modify the log level of various loggers. See [LAT Log4J Configuration](/lat-log4j-configuration) for descriptions of the loggers and log levels. ### `ROOT_LOG_LEVEL` | Logger: | root | | -------- | ----- | | Default: | debug | ### `BINDERS_LOG_LEVEL` | Logger: | com.ocient.lat.sink.ocient.Binders.Binder | | -------- | ----------------------------------------- | | Default: | info | ### `HTTP_LOG_LEVEL` | Logger: | org.eclipse.jetty.server.HttpChannel | | -------- | ------------------------------------ | | Default: | debug | ### `KAFKA_LOG_LEVEL` | Logger: | org.apache.kafka | | -------- | ---------------- | | Default: | info | ### Java The `JAVA_OPTS` environment variable is available for controlling settings. Container defaults to default JVM settings. See the [Java command line documentation](https://docs.oracle.com/en/java/javase/17/docs/specs/man/java.html#overview-of-java-options) for detail on the command line options. Example: ```shell Shell theme={null} JAVA_OPTS="-XX:MaxDirectMemorySize=32G -Xms32G -Xmx32G" ``` ### Data Volumes The LAT in the docker container writes persistent data to `/lat/data`. It is recommended to mount a volume to this path so that the data can outlive the container instance. The [Error Log File](/lat-advanced-topics#error-log-file) is also written to this path if configured. If viewing this error file is necessary, a volume must be used. Do not use the `LAT_DATA_PATH` configuration to override the data path inside the docker container. It will cause the LAT to write data and error logs into different locations and will require two volumes. If using the Local File Source, the user must mount the data directory as a bind mount inside the container. It is recommended to mount to a location under the `/lat/` directory. When configuring the local file source the bind mount path should be configured with a file group. ### Ports By default the LAT API listens on port `8080`. It is not recommended to override the `LAT_API_PORT` environment variable and instead use Docker’s port mapping functionality. ### Examples ### Docker ```shell Shell theme={null} docker run --rm --name lat \ --env ROOT_LOG_LEVEL=info \ --env JAVA_OPTS="-XX:MaxDirectMemorySize=32G -Xms32G -Xmx32G" \ -p 8080:8080 \ --mount source=lat-data,target=/lat/data \ lat:2.0.0 ``` ### Docker Compose ```yaml YAML theme={null} --- version: '3.9' services: lat: image: lat:2.0.0 ports: - "8080:8080" restart: on-failure volumes: - lat-data:/lat/data environment: JAVA_OPTS: "-XX:MaxDirectMemorySize=32G -Xms32G -Xmx32G" ROOT_LOG_LEVEL: info volumes: lat-data: ``` ## Related Links [Install an Ocient System](/install-an-ocient-system) [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Pipeline Configuration Source: https://docs.ocient.com/lat-pipeline-configuration Configure an Ocient LAT pipeline by defining sources, transforms, and sinks in a configuration file to control end-to-end data loading behavior. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). A pipeline configuration is a JSON file that fully describes the necessary elements to run a pipeline, including * **Source** - the source location from which a pipeline should read records to process * **Extract** - how a pipeline should extract data from the source * **Transform** - how a pipeline should transform incoming records * **Sink** - the destination where a pipeline should write transformed rows The JSON file is a list of keys with (possibly nested) values. The available key-value pairs are documented in these sections. ## Pipeline Top level configuration for a pipeline. Required keys: * [version](#version) * [source](#source) * [extract](#extract) * [transform](#transform) The following is an example of the structure of a pipeline configuration. ```json JSON theme={null} { "version": 2, "source": { "type": "kafka" // kafka source configuration }, "sink": { "type": "ocient" // ocient sink configuration }, "extract": { // extract configuration }, "transform": { "topics": { "my-topic": { "tables": { "my-table": { "columns": { "my-col1": "record_field_1", "my-col2": "record_field_2" } } } } } } } ``` ## Configuration ### `version` The pipeline’s version. Required value is `2`. | Type: | `int` | | --------- | ----- | | Required: | Yes | | Default: | | ### `pipeline_id` A unique identifier for this pipeline. Allowed characters are `a-z`, `A-Z`, `0-9`, `_`, and `-`. The `pipeline_id` is used to uniquely identify a pipeline. It is used for a few purposes: Deduplication scope. See [Understanding Deduplication](/lat-advanced-topics#understanding-deduplication). 1. For loads, the consumer `group.id` is set to `ocient-lat-[pipeline_id]`. For most loads from File Sources, it is advisable to leave the `pipeline_id` unset when creating a pipeline using the LAT client. The client will assign the pipeline a random UUID. | Type: | `string` | | --------- | ----------------------------------- | | Required: | No | | Default: | LAT client randomly generated UUID. | ### `workers` The number of workers this pipeline should use for processing records. | Type: | `int` | | --------- | ---------------------------------------------- | | Required: | No | | Default: | `DEFAULT_NUM_WORKERS` in service configuration | ### `log_original_records` Add original records to the error log when errors occur. When this setting is set to `true`, LAT writes data extracted from the source to the error log and in some error messages. By default, this setting is `false` and source data is not written to the error log nor included in error messages. In order to enable this setting, the [LAT\_ALLOW\_LOG\_ORIGINAL\_RECORDS](/lat-service-configuration#lat_allow_log_original_records) service configuration must also be enabled. This configuration only affects pipelines that do not use an [error\_topic](#error_topic). ### `seek_on_rebalance` Whether to seek a newly assigned partition to the latest known durable record prior to resuming processing. Disabling this behavior should typically be reserved for test scenarios and is only supported for Kafka loading. | Type: | `boolean` | | --------- | --------- | | Required: | No | | Default: | `true` | ### `continue_on_unrecoverable_error` Whether to allow workers to continue processing when they encounter an ordinarily unrecoverable error. | Type: | `boolean` | | --------- | --------- | | Required: | No | | Default: | `false` | ### `single_file_mode` Enable the single file mode. This mode is designed for a specific use case where there are few files but each file size is large. When using this mode, only a single file is processed at a time, so [Common File Source Configuration](/lat-source-configuration#common-file-source-configuration) must be equal to `1` and only one [Common File Source Configuration](/lat-source-configuration) can be defined. The single file is processed in parallel by the number of workers defined by the pipeline `workers` setting. There is no need to enable this in common use cases. | Type: | `boolean` | | --------- | --------- | | Required: | No | | Default: | `false` | A known limitation exists with the LAT metrics when you use the single file mode. Metrics returned from the `lat_client pipeline status` command might not display the expected count of files processed, processing, and so on. However, the load still displays the correct `PROCESSING` and `COMPLETED` statuses. ### `error_topic` A Kafka Topic to write records which cannot be processed. If absent, error records will be logged to the error log file without additional processing. This configuration is only available if a Kafka Source is configured for the pipeline. The configuration for that source will apply to the Kafka Producer for this topic. | Type: | `string` | | --------- | -------- | | Required: | No | | Default: | `null` | ### `polling_duration` Maximum duration to block while polling for new records from a Source, in milliseconds. | Type: | `int` | | --------- | ------ | | Required: | No | | Default: | `1000` | ### `source` Source configuration section. See [LAT Source Configuration](/lat-source-configuration) for nested configuration. | Type: | `object` | | --------- | -------- | | Required: | Yes | | Default: | | ### `sink` Sink configuration section. See [LAT Sink Configuration](/lat-sink-configuration) for inline configuration details. `sink` cannot be set if `sink_name` is set. `sink` can be omitted if a default sink is defined as an [External Sink Configuration](/lat-sink-configuration#external-sink-configuration). | Type: | `object` | | --------- | -------- | | Required: | No | | Default: | `null` | ### `sink_name` The name of an externally configured sink. `sink_name` cannot be set if [sink](#sink) is set. See [External Sink Configuration](/lat-sink-configuration). `sink_name` can be omitted if a default sink is defined as an [External Sink Configuration](/lat-sink-configuration). | Type: | `string` | | --------- | -------- | | Required: | No | | Default: | `null` | ### `extract` Extract configuration section. See [LAT Extract Configuration](/lat-extract-configuration) for nested configuration. | Type: | `object` | | --------- | -------------------------------------------------- | | Required: | No | | Default: | Defaults to JSON record type with default settings | ### `transform` Transform configuration section. See [LAT Transform Configuration](/lat-transform-configuration) for nested configuration. | Type: | `object` | | --------- | -------- | | Required: | Yes | | Default: | | ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Record Filtering Source: https://docs.ocient.com/lat-record-filtering Filter records in Ocient LAT pipelines using expression-based rules to drop, route, or accept incoming records before they reach transformation and load stages. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). Incoming records can optionally be filtered by a JMESPath expression added with an additional filter key in a pipeline configuration. ```json JSON theme={null} { "transform": { "topics": { "my_topic": { "filter": "", "tables": { "my_table": { "filter": "", "columns": {} } } } } } } ``` Records that pass through the filter (the filter expression evaluates to true) are processed as normal, while records that fail the filter (the filter expression evaluates to false) are dropped. Note that filters can be applied at the topic or table levels. A record needs to pass through all configured filters in order to be loaded to a table. The filter JMESPath expression must evaluate to a `boolean` value. Examples of this are JMESPath comparators or functions that return a `boolean` like `contains`, `starts_with`, and `ends_with`. ## Example Pipeline: ```json JSON theme={null} { "transform": { "topics": { "topic0": { "filter": "integer0 == `2`", "tables": { "table0": { "filter": "integer1 == `3`", "columns": { "col0": "integer0", "col1": "integer1" } } } } } } } ``` Input: ```json JSON theme={null} [ { "integer0": 0, "integer1": 3 }, { "integer0": 2, "integer1": 1 }, { "integer0": 2, "integer1": 3 } ] ``` Output: ```json JSON theme={null} [ { "integer0": 2, "integer1": 3 } ] ``` ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Service Configuration Source: https://docs.ocient.com/lat-service-configuration Configure the Ocient LAT service, including JVM options, threading, resource allocation, ports, and other parameters that affect pipeline runtime behavior. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). The LAT service is configurable using Service Configuration. This configuration is derived from three sources, with the following precedence order: 1. Environment Variables 2. Properties File 3. Code Defaults ## Configuration File Backup When you install a new LAT instance or upgrade the existing instance, LAT creates a backup file of the `/etc/lat/lat.conf` as `/etc/lat/override.conf` and creates a new version of the `/etc/lat/lat.conf` file. The `/etc/lat/overrirde.conf` file does not control the LAT configuration. If you made prior changes to `lat.conf`, you must review `override.conf` and add any custom values into the revised `lat.conf`. ## Environment Variables ### `LAT_DEFAULT_WORKERS` Default number of workers used by a newly configured pipeline. | **Environment Variable:** | LAT\_DEFAULT\_WORKERS | | ------------------------- | --------------------- | | **Property**: | lat.default.workers | | **Default**: | Server core count | ### `LAT_SHUTDOWN_DURATION` Duration, in milliseconds, to wait prior to termination on shutdown. | **Environment Variable:** | LAT\_SHUTDOWN\_DURATION | | ------------------------- | ----------------------- | | **Property**: | lat.shutdown.duration | | **Default**: | 120000 | ### `LAT_UDT_PATH` Directory containing zero or more user defined transformation (UDT) jars. Ensure that the specified path can be accessed by the LAT (e.g., does not allow the LAT to access the home directory of an administrative user). | **Environment Variable:** | LAT\_UDT\_PATH | | ------------------------- | -------------- | | **Property**: | lat.udt.path | | **Default**: | null | ### `LAT_API_PORT` REST API and Metrics port. | **Environment Variable:** | LAT\_API\_PORT | | ------------------------- | -------------- | | **Property**: | lat.api.port | | **Default**: | 8080 | ### `LAT_DATA_PATH` Path to the LAT data directory. This directory is used to store persistent data related to pipelines (`pipeline/`) and sinks (`sink/`). In almost all cases, using the [LAT Client Command Line Interface](/lat-client-command-line-interface) is preferred over editing the files in this directory manually. Ensure that the specified path can be accessed by the LAT (e.g., CentOS does not allow the LAT to access the home directory of an administrative user). | **Environment Variable:** | LAT\_DATA\_PATH | | ------------------------- | --------------- | | **Property**: | lat.data.path | | **Default**: | .lat-data | ### `LAT_AUTO_START` On LAT startup, automatically starts running a Pipeline if one is present. This only supports pipelines with sources; File sources will not automatically start even if this configuration is set to `true`. | **Environment Variable:** | LAT\_AUTO\_START | | ------------------------- | ---------------- | | **Property**: | lat.auto.start | | **Default**: | false | ### `LAT_ALLOW_LOG_ORIGINAL_RECORDS` Allow the original records that are being loaded to be logged when errors occur. This setting must be enabled for `log_original_records` to be enabled. This setting only affects pipelines that log errors to the error log file. | **Environment Variable:** | LAT\_ALLOW\_LOG\_ORIGINAL\_RECORDS | | ------------------------- | ---------------------------------- | | **Property**: | lat.allow\.log.original.records | | **Default**: | false | ### `LAT_MAX_RETRIEVABLE_ERRORS` A limit on the number of errors that you can retrieve using the `pipeline errors` command of the LAT client. | **Environment Variable:** | LAT\_MAX\_RETRIEVABLE\_ERRORS | | ------------------------- | ----------------------------- | | **Property**: | lat.max.retrievable.errors | | **Default**: | 1000 | ### `LAT_API_PROXY_HOST` API proxy host for authorization requests. | **Environment Variable:** | LAT\_API\_PROXY\_HOST | | ------------------------- | --------------------- | | **Property**: | lat.api.proxy.host | | **Default**: | null | ### `LAT_API_PROXY_PORT` API proxy port for Okta authorization requests. | **Environment Variable:** | LAT\_API\_PROXY\_PORT | | ------------------------- | --------------------- | | **Property**: | lat.api.proxy.PORT | | **Default**: | null | ### `LAT_API_OAUTH_DOMAIN` Okta authorization domain. | **Environment Variable:** | LAT\_API\_OAUTH\_DOMAIN | | ------------------------- | ----------------------- | | **Property**: | lat.api.oauth.domain | | **Default**: | null | ### `LAT_API_OAUTH_ENDPOINTS` A list of LAT API endpoints that Okta should apply authentication on. To override, provide a comma-delimited list, e.g., `/endpoint1,/endpoint2,/endpoint3`. | **Environment Variable:** | LAT\_API\_OAUTH\_ENDPOINTS | | ------------------------- | -------------------------- | | **Property**: | lat.api.oauth.endpoints | | **Default**: | null | ### `LAT_API_OAUTH_SERVER` Okta authorization server. | **Environment Variable:** | LAT\_API\_OAUTH\_SERVER | | ------------------------- | ----------------------- | | **Property**: | lat.api.oauth.server | | **Default**: | null | ### `LAT_API_CERT_PATH` API cert file path. Ensure that the specified path can be accessed by the LAT (e.g., CentOS does not allow the LAT to access the home directory of an administrative user). | **Environment Variable:** | LAT\_API\_CERT\_PATH | | ------------------------- | -------------------- | | **Property**: | lat.api.cert.path | | **Default**: | null | ### `LAT_DNS_CACHE_TTL` The time-to-live (TTL) in seconds that the LAT should use for its DNS cache. This configuration is only supported as an Environment Variable or as Default. | **Environment Variable:** | LAT\_DNS\_CACHE\_TTL | | ------------------------- | -------------------- | | **Property**: | N/A | | **Default**: | 30 | ### `LAT_MEMORY` Maximum heap memory. This configuration is only supported as an Environment Variable or as Default when running the LAT through the provided package service file or through the provided `lat.sh` script. If directly running the Jar JVM heap memory should be set through command line arguments. | **Environment Variable:** | LAT\_MEMORY | | ------------------------- | ----------- | | **Property**: | N/A | | **Default**: | 32G | ### `LAT_MEMORY_DIRECT` Maximum JVM direct memory. This configuration is only supported as an Environment Variable or as Default when running the LAT through the provided package service file or through the provided `lat.sh` script. If directly running the Jar JVM heap memory should be set through command line arguments. | **Environment Variable:** | LAT\_MEMORY\_DIRECT | | ------------------------- | ------------------- | | **Property**: | N/A | | **Default**: | 32G | ## Override Environment Variables The recommended way to manage the production service configuration when LAT is running as a daemon is using environment variables or the `/etc/systemd/system/lat.service.d/override.conf` file. To override LAT environment variables as a `systemd` service, there are two recommended commands that edit the `etc/systemd/system/lat.service.d/override.conf` file. 1. Run `sudo vim etc/systemd/system/lat.service.d/override.conf` or `sudo systemctl edit lat`. 2. Insert the environment variables to override in the file. Save and exit the editor. **Configuration File Example:** ```shell Shell theme={null} [Service] Environment=LAT_MEMORY_DIRECT=200G Environment=LAT_MEMORY=200G ``` ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) [LAT Packaging and Installation](/lat-packaging-and-installation) # LAT Sink Configuration Source: https://docs.ocient.com/lat-sink-configuration Configure Ocient LAT pipeline sinks to write transformed data into Ocient tables or other targets, including batching, commit, and error-handling options. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). The Sink Configuration controls the destination of data in the LAT Pipeline. ## `sink` A sink configuration object. Required keys: * [sink.type](#sink-type) ### `sink.type` \[#sink-type] Type of sink to use in the pipeline. | **Type:** | string | | ------------- | ------ | | **Required**: | Yes | | **Default**: | | Allowed values: * `ocient`: see [Ocient Sink](#ocient-sink) for additional configuration. * `file`: see [File Sink](#file-sink) for additional configuration. ### Ocient Sink The Ocient Sink allows LAT to connect to an Ocient cluster to write rows to one or more tables. Required keys: * [sink.remotes](#sink-remotes) #### `sink.remotes` \[#sink-remotes] Array of one or more Ocient Loader Nodes, in `host:port,...` format | **Type**: | string\[] | | ------------- | --------- | | **Required**: | Yes | | **Default**: | | #### `sink.batch_records` Number of records to buffer per partition before flushing records to Ocient | **Type**: | int | | ------------- | ---- | | **Required**: | No | | **Default**: | 1000 | #### `sink.batch_duration` Time based flushing parameter, in milliseconds. Records will flush to Ocient after this duration has elapsed with no new activity, even if fewer than `batch_records` records have been processed. | **Type**: | int | | ------------- | ----- | | **Required**: | Yes | | **Default**: | 30000 | #### `sink.idle_partition_polling_period` Time based polling parameter, in milliseconds. This Sink will periodically poll the remote for progress on write durability for idle partitions. | **Type**: | int | | ------------- | ----- | | **Required**: | No | | **Default**: | 60000 | #### `sink.request_timeout` Request timeout when communicating with Ocient remotes, in milliseconds. | **Type**: | int | | ------------- | ------ | | **Required**: | No | | **Default**: | 300000 | #### `sink.request_backoff` Duration to delay after a failed request to an Ocient remote prior retrying, in milliseconds. | **Type:** | int | | ------------- | ---- | | **Required**: | No | | **Default**: | 1000 | #### `sink.request_jitter` Additional duration to delay after a failed request to an Ocient remote prior retrying, in milliseconds. The total delay incurred prior to a given retry is `request_backoff + rand(0, request_jitter)`. | **Type**: | int | | ------------- | ---- | | **Required:** | No | | **Default**: | 5000 | #### `sink.high_watermark` High watermark memory point, in bytes. The LAT will stop pushing new rows to memory buffers. It will not resume pushing rows into the memory buffers until `low_watermark` is reached. | **Type**: | int | | ------------- | ---------- | | **Required**: | No | | **Default**: | 1000000000 | #### `sink.low_watermark` Low watermark memory point, in bytes. After reaching `high_watermark`, the LAT will begin pushing rows to memory buffers again when this memory level is reached. | **Type**: | int | | ------------- | --------- | | **Required**: | No | | **Default**: | 500000000 | #### `sink.storage_scope_id` UUID of the storage scope that rows will be associated with. The scope with the given UUID must already exist in the target cluster. | **Type**: | string | | ------------- | ------ | | **Required**: | No | | **Default**: | `null` | #### `sink.skip_page_replication` A Boolean value to determine whether to omit page replicas for the specified storage scope. This is ignored if `sink.storage_scope_id` is not specified or has already been seen by the remotes. | **Type**: | boolean | | ------------- | ------- | | **Required**: | No | | **Default**: | `false` | #### `sink.netty_event_loop_group_threads` The number of threads in the Netty event loop group used to communicate with remotes. | **Type**: | int | | ------------- | --- | | **Required**: | No | | **Default**: | `1` | #### Example Ocient Sink Configuration ```json JSON theme={null} { "sink": { "type": "ocient", "remotes": ["loader0:5050", "loader1:5050", "loader2:5050"] } } ``` ### File Sink A Sink type for **testing** LAT pipelines that writes the transformed data to local [JSONL](https://jsonlines.org/) files. Required keys: * [sink.location](#sink-location) #### sink.location \[#sink-location] An absolute or relative path to the location that the sink should write files to. | **Type**: | String | | ------------- | ------ | | **Required**: | Yes | | **Default**: | | #### Example File Sink Configuration ```json JSON theme={null} { "sink": { "type": "file", "location": "out/" } } ``` ## External Sink Configuration Rather than including a sink directly within the pipeline, it is also possible to configure a pipeline to use a sink that is specified externally. Sinks can be managed (created, deleted, and more) using the [LAT Client Command Line Interface](/lat-client-command-line-interface). A sink must exist before a pipeline can use it. There are three ways to configure a pipeline to use a sink. 1. If a sink is included directly within a pipeline (using the [LAT Sink Configuration](#lat-sink-configuration)), it will be used. 2. If a sink is not specified within the pipeline, you can specify a [sink\_name](/lat-pipeline-configuration#sink_name) that corresponds to a sink previously created using the LAT Client. 3. If neither `sink` nor `sink_name` is specified in a pipeline, the default sink will be used. If a default sink has not been created using the LAT Client, a pipeline must specify either a `sink` or a `sink_name`. ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Source Configuration Source: https://docs.ocient.com/lat-source-configuration Configure Ocient LAT pipeline sources for loading data from Kafka, S3, file systems, and other systems, including connection details and parser settings. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). ## Source A source configuration object. Required keys: * [source.type](#source-type) ### `source.type` \[#source-type] Type of source to use in the pipeline. | **Type:** | string | | ------------- | ------ | | **Required:** | Yes | | **Default:** | | Allowed values: * `kafka`: see [Kafka Configuration](#kafka-configuration) for additional configuration. * `s3`: see [Load from a File Source](#load-from-a-file-source) and [S3 Source Configuration](#s3-source-specific-configuration)  for additional configuration. * `local`: see [Load from a File Source](#load-from-a-file-source) for additional configuration. ## Load from a File Source Currently, LAT supports loading files from an S3 instance (like AWS S3 or Object Storage) or from a local file system. Common configuration for all File Sources will be listed in [Common File Source Configuration](#common-file-source-configuration), followed by sections describing source-specific configurations. File Sources are defined by creating "file groups" that represent a logical set of files to be loaded. Each file group is given a `file_group_name` that corresponds to the file group name in the transform section of the pipeline configuration. Each file group has settings to select the specific files that should be loaded. The LAT supports loading from individual files with extensions such as `.jsonl` and `.csv`. GZIP compression is supported. However, files using the TAR archive format, LZOP compression, or ZIP compression are not supported. If you load files with unsupported archive or compression formats, the load might stop or produce unexpected results. The performance of loading into Ocient is greatly improved when records are presented in a well ordered time sequence. This sequence allows more efficient creation of segments and sorting of records into buckets. For this reason, the LAT has options to define how files in a file group should be sorted prior to loading. ### File Group Filtering and Sorting Example If you want to load all the files under a certain directory, the file group configuration is basic: ```json JSON theme={null} "my_file_group": { "prefix": "/dir1", "file_matcher_syntax": "glob", "file_matcher_pattern": "**", "sort_type": "lexicographic" } ``` This file group configuration will select all files under `/dir1` (including those under its subdirectories) and sort them lexicographically. However, if a user needs to selectively choose, filter, and sort the files selected, this process provides flexible options for doing so. For an individual file group in a file type load, there are multiple steps to select, rename, filter, and sort the files retrieved. The files selected occur through this sequence: 1. Prefix filtering 2. File matching 3. File renaming 4. Timestamp extract pattern matching (only `extract_timestamp`) 5. Range matching 6. Final list sorting For example, given these files from a local file system: ```Text Text theme={null} /dir1/file.json /dir1/files1/year=2021/month=01/day=01/files2/10-00-00.csv /dir1/files1/year=2021/month=01/day=01/file.json /dir1/files1/year=2021/month=01/day=01/files3/11-00-00.json /dir1/files1/year=2021/month=01/day=01/files2/12-00-00.json /dir1/files1/year=2021/month=01/day=01/files2/14-00-00.json /dir1/files1/year=2021/month=01/day=01/files2/13-00-00.json ``` And a `file_group` configuration: ```json JSON theme={null} "my_group": { "prefix": "/dir1/files1/", "file_matcher_syntax": "regex", "file_matcher_pattern": "/dir1/files1/year=(\\d{4})/month=(\\d{2})/day=(\\d{2})/files\\d/(\\d{2})-(\\d{2})-(\\d{2}).json", "rename_format": "{1}-{2}-{3}-{4}-{5}-{6}", "sort_type": "extract_timestamp", "path_timestamp_pattern": "yyyy-MM-dd-HH-mm-ss", "start_time": "2021-01-01T13:00:00", "stop_time": "2021-01-01T14:00:01" } ``` **Step 1:** Prefix Filtering Prefix filtering occurs first and includes only files that are in the matching prefix paths. The prefix is the path part following the bucket for S3 types, and it is the path to files in a local file type load. ```json JSON theme={null} "prefix": "/dir1/files1/" ``` Result: `/dir1/file.json` is filtered out because it is not in the `prefix`. **Step 2:** File Matching Next, file matching uses a pattern to match files that should be included. File matcher patterns apply to the full path of the file including any prefix defined in the prior step. ```json JSON theme={null} "file_matcher_syntax": "regex" "file_matcher_pattern": "/dir1/files1/year=(\\d{4})/month=(\\d{2})/day=(\\d{2})/files\\d/(\\d{2})-(\\d{2})-(\\d{2}).json" ``` Result: `/dir1/files1/year=2021/month=01/day=01/files2/10-00-00.csv` is filtered out because it does not end in `.json`. `/dir1/files1/year=2021/month=01/day=01/file.json` is filtered out because it is not in the subdirectory `files1`. **Step 3:** File Renaming File renaming is a step that can be useful in a case where the selected files have disparate file names that would make it difficult to extract timestamps from. Files are only renamed internally to the LAT to facilitate the file selection process; they are *not* actually renamed locally or on S3. By unifying each filename into a consistent format, this step should make it easier to use the `extract_timestamp` sort type, or even to `lexicographically` sort on some parameter within the filename itself. ```json JSON theme={null} "file_matcher_pattern": "/dir1/files1/year=(\\d{4})/month=(\\d{2})/day=(\\d{2})/files\\d/(\\d{2})-(\\d{2})-(\\d{2}).json" "rename_format": "{1}-{2}-{3}-{4}-{5}-{6}", ``` Result: The files that match the `file_matcher_pattern` regular expression are renamed using the `rename_format`. Current Files at this Step: ```Text Text theme={null} /dir1/files1/year=2021/month=01/day=01/files3/11-00-00.json /dir1/files1/year=2021/month=01/day=01/files2/12-00-00.json /dir1/files1/year=2021/month=01/day=01/files2/14-00-00.json /dir1/files1/year=2021/month=01/day=01/files2/13-00-00.json ``` Renamed file list that enters the next step: ```Text Text theme={null} 2021-01-01-11-00-00 2021-01-01-12-00-00 2021-01-01-14-00-00 2021-01-01-13-00-00 ``` **Step 4:** Timestamp Extract Pattern Matching A filter is applied when the sort\_type is `extract_timestamp`. The `extract_timestamp` sort type extracts timestamps from the file’s filepath. Either a `path_timestamp_pattern` or a `file_timestamp_pattern` must be set. If a filename does not match the set pattern, it will be filtered out. ```json JSON theme={null} "sort_type": "extract_timestamp" "path_timestamp_pattern": "yyyy-MM-dd-HH-mm-ss", ``` Result: In this example, a `path_timestamp_pattern` is set, which is a [`DateTimeFormatter`](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html) pattern that will extract a timestamp from the starting from the beginning of the filename, used for the next step. No files are filtered here because the rename step above was able to unify filename formats in a way that the `path_timestamp_pattern` could match on all the files. Internal to the LAT, the files are associated with their datetimes, which will be used in the next step: ```Text Text theme={null} filename -> datetime extracted in ISO 8601 format 2021-01-01-11-00-00 -> 2021-01-01T11:00:00Z 2021-01-01-12-00-00 -> 2021-01-01T12:00:00Z 2021-01-01-14-00-00 -> 2021-01-01T14:00:00Z 2021-01-01-13-00-00 -> 2021-01-01T13:00:00Z ``` There are potential pitfalls with using the `extract_timestamp` `sort_type`, described in the [LAT Source Configuration](#page-title) section. **Step 5:** Range Matching Range matching occurs next based on the chosen `sort_type` algorithm. * For `extract_timestamp` or `metadata`, a timestamp is associated with each file for sorting. A start and end time can be optionally provided to limit files to select. * For `lexicographic`, the filename itself is used for sorting. A start and end filename can be provided to limit the files that are selected. ```none Text theme={null} "sort_type": "extract_timestamp" "path_timestamp_pattern": "yyyy-MM-dd-HH-mm-ss", "start_time": "2021-01-01T13:00:00" "stop_time": "2021-01-01T14:00:01"t ``` Result: The file `2021-01-01-12-00-00` is filtered out because it is not in the start and end time ranges. **Step 6:** Final List Sorting Finally, the fully filtered file list contains two files from your original list. These are then sorted according to the `sort_type` algorithm and partitioned across the workers for loading. Remaining files before sorting: ```Text Text theme={null} 2021-01-01-14-00-00 2021-01-01-13-00-00 ``` Final list - sorted, renamed files: ```Text Text theme={null} 2021-01-01-13-00-00 2021-01-01-14-00-00 ``` (original file names): ```Text Text theme={null} /dir1/files1/year=2021/month=01/day=01/files2/13-00-00.json /dir1/files1/year=2021/month=01/day=01/files2/14-00-00.json ``` ### Example S3 Source Configuration The names defined for `file_groups` ("my\_file\_group" in this example) should match the `file_groups` used in the transform configuration section. ```json JSON theme={null} { "source": { "type": "s3", "endpoint": "http://some.endpoint/", "bucket": "my_bucket", "file_groups": { "my_file_group": { "prefix": "some/prefix/", "file_matcher_syntax": "glob", "file_matcher_pattern": "**.json", "sort_type": "extract_timestamp", "path_timestamp_pattern": "'dir1/files1/'yyyy'-'MM'-'dd'T'HH':'mm':'ss", } }, "compression": "gzip" }, "transform" : { "file_groups" : { "my_file_group" : { "tables" : { ... } } } } } ``` ### Example Local File Source Configuration The names defined for `file_groups` ("my\_file\_group" in this example) should match the `file_groups` used in the transform configuration section. ```json JSON theme={null} { "source": { "type": "local", "file_groups": { "my_file_group": { "prefix": "/path/to/data/directory/", "file_matcher_syntax": "glob", "file_matcher_pattern": "**.json", "sort_type": "extract_timestamp", "path_timestamp_pattern": "'dir1/files1/'yyyy'-'MM'-'dd'T'HH':'mm':'ss", } }, "compression": "gzip" }, "transform" : { "file_groups" : { "my_file_group" : { "tables" : { ... } } } } } ``` ### Common File Source Configuration `source.file_groups` An object that maps file group names to their corresponding configuration objects. | Type: | object | | --------- | ------ | | Required: | Yes | | Default: | | `source.file_groups.` The configuration object for a file group. | Type: | object | | --------- | ------ | | Required: | Yes | | Default: | | `source.file_groups..prefix` For an S3 Source, the `prefix` is used to get a subset of S3 objects from the bucket. A more specific prefix can improve time required to list files in S3 sources. For a Local File Source, `prefix` is an absolute or relative path from the working directory of the LAT. In either case, the path can be a file or a directory. If the path is a directory, LAT tries to load all files recursively under this path. When looking for files, LAT follows the symbol links. LAT only loads regular files that are readable and not hidden. Note that shell-specific expansions like `~` are not supported. You can include multiple prefixes for a single file group if you specify this property as an array of strings. In that case, the LAT loads all files that are under any of the prefixes in that array. The system loads files that match more than one prefix within a single file group only once. For a Local File Source, the path used for prefix is the path on the server where the LAT is running, which might not be the same machine where you are running the LAT Client Command Line Interface to create your pipeline. | Type: | string or string array | | --------- | ---------------------- | | Required: | No | | Default: | `""` | `source.file_groups..file_matcher_syntax` The `file_matcher_syntax` defines the type of syntax used by the `file_matcher_pattern` setting, which includes matching files in the file group. The pattern is applied to the fully qualified filename in the list of files found under the `prefix`. See the `file_matcher_pattern` description for examples and complete syntax details for `glob` and `regex` options. | Type: | string | | --------- | ------------------------------------------------------------------------------------------------------------ | | Required: | No, although it is not valid to have only one of `file_matcher_syntax` or `file_matcher_pattern` be provided | | Default: | `glob` | Allowed values: * `glob` - a simplified pattern matching system based on wildcards * `regex` - a regular expression syntax `source.file_groups..file_matcher_pattern` The `pattern` used to select files from the `prefix`-filtered list. Files that match the pattern are included in the file group. The **fully qualified filename** (path and filename under the prefix) are matched. Files that do not match are excluded from the file group. This pattern is also used in the case of LAT **file renaming** - if a `rename_format` is provided, the LAT will attempt to internally rename files using capture groups in the `file_matcher_pattern`. If capture groups are provided in the `file_matcher_pattern`, they will be extracted and placed into the `rename_format` in order to generate a renamed file. Capture groups do *not* need to be named; the captured values are assigned to the `rename_format` sequentially, i.e. the first capture group in the `file_matcher_pattern` will be placed in `{1}` in the `rename_format`. Files are only renamed internally to the LAT to facilitate the file selection process; they are *not* actually renamed locally or on S3. This matched file list can be further filtered using an `extract_timestamp's` pattern and the start/end ranges of any file group type algorithm. See an expanded example in the [File Group Filtering and Sorting example](#file-group-filtering-and-sorting-example) section. A double backslash within the regular expression is necessary to generate valid escape characters for the `pipeline.json`. For example, the capture group `(\d{4})` needs to be escaped as `(\\d{4})`. | Type: | string | | --------- | ------------------------------------------------------------------------------------------------------------ | | Required: | No, although it is not valid to have only one of `file_matcher_syntax` or `file_matcher_pattern` be provided | | Default: | `**` | **Example** With a `file_matcher_syntax` of `glob` and `file_matcher_pattern` of `**.json`, all files (after `prefix`-filtering) that end in `.json` will be selected. A `**` in the glob based pattern ensures that any pattern including directory boundaries are matched. A pattern `*` does not match the directory character `/`. **Example** With a `file_matcher_syntax` of `regex`, a `file_matcher_pattern` of `auctions.*-2021.json` would use a regular expression to ensure that files such as `auctions-12-01-2021.json` and `auctions-12-02-2021.json` would be selected. See [`getPathMatcher()`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/FileSystem.html#getPathMatcher\(java.lang.String\)) for complete syntax details of the `glob` and `regex` options. Common Matcher Patterns: | **Matcher Syntax** | **Pattern** | **Meaning** | | ------------------ | ----------- | ---------------------------------------------------------------------------------------- | | glob | `*` | Matches zero or more characters without crossing directory boundaries | | glob | `**` | Matches zero or more characters crossing directory boundaries | | glob | `?` | Matches exactly one character of a name component | | glob | `[]` | Matches any of the characters in the bracket expression (e.g., \[abc]). Supports ranges. | | regex | `.` | Matches any character | | regex | `\d` | Matches any digit | | regex | `\D` | Matches any non-digit | | regex | `[]` | Matches any of the characters in the bracket expression (e.g., \[abc]). Supports ranges. | | regex | `*` | Matches the preceding character zero or more times | | regex | `+` | Matches the preceding character one or more times | | regex | `?` | Matches the preceding character once or not at all | `source.file_groups..rename_format` A [`MessageFormat`](https://docs.oracle.com/javase/7/docs/api/java/text/MessageFormat.html) string. Captured groups from the `file_matcher_pattern` will be placed into the `rename_format` string’s format elements corresponding to the order they were captured. The renamed file is used for subsequent sorting of files. When using `lexicographic` sort, the renamed file is used instead of the original filename. For `extract_timestamp` sorting, the `path_timestamp_pattern` is applied against the renamed file. This is best elucidated with an example: ```Text Text theme={null} filename: 20210102.json file_matcher_pattern: (\\d{4})(\\d{2})(\\d{2}).* rename_format: "file-{1}-{3}-{2}" "sort_type": "extract_timestamp" "path_timestamp_pattern": "'file-'yyyy'-'MM'-'dd" Renames the file to: "file-2021-02-01" The "path_timestamp_pattern" extracts an ISO-8601 datetime of 2021-02-01T00:00:00Z from the renamed file ``` Files are only renamed internally to the LAT to facilitate the file selection process; they are *not* actually renamed locally or on S3. | Type: | string | | --------- | -------------------------------------------------------------------------------- | | Required: | No | | Default: | None - if `rename_format` is not provided, files will not undergo a rename step. | `source.file_groups..sort_type` Defines the sorting algorithm used for sorting the files selected for loading in this file group by time. Files should be sorted in time order for best loading performance and the creation of efficient Ocient segments. `sort_type` can be `extract_timestamp`, `metadata`, or `lexicographic`, and different settings apply to each of these selections. | Type: | string | | --------- | ------ | | Required: | Yes | | Default: | `none` | Allowed values: * `extract_timestamp`: extract the timestamp from the filename or file path information. Either `path_timestamp_pattern` or `file_timestamp_pattern` is required when choosing this `sort_type`. * `metadata`: extract the timestamp from the source file’s metadata. This option uses the file last modified time when sorting the list. * `lexicographic`: sort files based on the alphanumeric sort of files in the file group using the full path and filename as the sorting key. The system breaks the tie among files in a lexicographic way by using their fully qualified original file names. `source.file_groups..path_timestamp_pattern` A Format string with [`DateTimeFormatter`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/time/format/DateTimeFormatter.html) patterns used to extract a datetime from the path portion of a file’s fully-qualified filename. This timestamp is used with the `extract_timestamp` sort type to order the selected files in the file group prior to loading. It is attempted on every file’s fully-qualified filename under the `prefix`; files that do not match the pattern will be skipped and the remaining subset will be used for sorting. | Type: | string | | --------- | ----------------------------------------------------------------------------------------------------------------------- | | Required: | One of `path_timestamp_pattern` or `file_timestamp_pattern` is required for an `extract_timestamp` sort type file group | | Default: | `none` | `source.file_groups..file_timestamp_pattern` A Format string with [`DateTimeFormatter`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/time/format/DateTimeFormatter.html) patterns used to extract a datetime from the base filename of a file’s fully-qualified filename. This timestamp is used with the `extract_timestamp` sort type to order the selected files in the file group prior to loading. It is attempted on every file’s base filename under the `prefix`; files that do not match the pattern will be skipped and the remaining subset will be used for sorting. | Type: | string | | --------- | ----------------------------------------------------------------------------------------------------------------------- | | Required: | One of `path_timestamp_pattern` or `file_timestamp_pattern` is required for an `extract_timestamp` sort type file group | | Default: | `none` | `source.file_groups..start_time` An ISO-8601 compliant Date or Datetime, used as the lower bound to filter for files in a `extract_timestamp` or `metadata` sort type file group. Datetimes are UTC unless a timezone is provided. Inclusive. | Type: | string | | --------- | ------------- | | Required: | No | | Default: | `Instant.MIN` | `source.file_groups..stop_time` An ISO-8601 compliant Date or Datetime, used as the upper bound to filter for files in a `extract_timestamp` or `metadata` sort type file group. Datetimes are UTC unless a timezone is provided. Exclusive. | Type: | string | | --------- | ------------- | | Required: | No | | Default: | `Instant.MAX` | \*Extract Timestamp Default Values \* With an `extract_timestamp` `sort_type`, if a certain time unit is not provided, they will default to their respective value in \`1970-01-01T00:00:00.000000\`. **Example:** ```none Text theme={null} Filename: `July03.json` `path_timestamp_pattern` or `file_timestamp_pattern`: `MMMMdd'.json'` ``` The full timestamp extracted from this filename will be `1970-07-03T00:00:00.000000 UTC`. *** **Potential Pitfalls** Pitfall 1: This default value should be considered when a `start_time` or `stop_time` is provided but a certain time unit is not present within the filename. **Example:** ```Text Text theme={null} Capture files ranging from July 1st through July 3rd: Filenames: July01.json, July02.json, July03.json "file_timestamp_pattern": "MMMMdd'.json'" "start_time": "1970-07-01" "stop_time": "1970-07-04" ``` Note that 1970 was used as the year in the `start_time` and `stop_time`, but the system could not extract a year from the filenames. An alternative strategy is to use the File Renaming step to insert missing dates or times. Pitfall 2: Only applicable to using the `extract_timestamp` `sort_type` with `start_time` and `stop_time`: Consider the time unit granularities between your pattern and start/stop times. **Example:** ```none Text theme={null} Filenames: dir1/2021/01/01/00:00:00.json, dir1/2021/01/01/01:00:00.json, dir1/2021/01/01/05:00:00.json "path_timestamp_pattern": "'dir1/'yyyy'/'MM'/'dd'/'" "start_time": "2021-01-01T00:00:00" "stop_time": "2021-01-01T04:00:00" ``` Given this configuration, one might expect that the files that are hour 4 and later would be filtered out. However, because the pattern does not extract the times from the filenames, each file’s extracted datetime in ISO 8601 format is `2021-01-01T:00:00:00UTC`, so no files are outside the range and filtered. Make sure to extract as much information about the datetime as necessary. `source.file_groups..start_file` A string, used as the lower bound to filter for files in a `lexicographic` sort type file group. Inclusive. | Type: | string | | --------- | ----------------------------------------------------------------------------------------------------------- | | Required: | No | | Default: | `none`, but a `lexicographic` sort type file group will not check for a lower bound if this is not present. | `source.file_groups..stop_file` A string, used as the upper bound to filter for files in a `lexicographic` sort type file group. Inclusive. | Type: | string | | --------- | ------------------------------------------------------------------------------------------------------------ | | Required: | No | | Default: | `none`, but a `lexicographic` sort type file group will not check for an upper bound if this is not present. | `source.file_groups..compression` The compression method for the files in this file group. If this value is set, it will override `source.compression`; if not set, it will inherit `source.compression`. See `source_compression` for available options. | Type: | string | | --------- | ------ | | Required: | No | | Default: | `null` | `source.file_groups..bucket` This setting is available only for S3 sources. If specified, this setting overrides the source-wide bucket for this file group. This allows loading from multiple buckets at once within the same pipeline. If a `bucket` is specified for every file group, you can omit the `source.bucket`. | Type: | string | | --------- | ------ | | Required: | No | | Default: | `null` | `source.partitions` The number of partitions for each file group. For example, if there are 2 file groups and `partitions` is set to 32, then a total of 64 partitions will exist, with 32 for each file group. The partition index is 0-based for each file group. In the above example, each file group will have partitions `[0, 31]`. When `partitions` is specified for a load, its value must not change for the lifetime of this load, otherwise records cannot load correctly. When using the LAT Client, `partitions` will be automatically assigned by the client. If it is set in the source configuration provided by the user, it will be overwritten. | Type: | int | | --------- | --------------------- | | Required: | No | | Default: | Set by the LAT client | `source.partitions_assigned` A 2-element array indicating the minimum and maximum partition indices that this LAT Node owns, inclusive on both ends. For example, `[0, 15]` means that the node owns Partitions 0, 1, 2, …, 15. The first element should be greater than or equal to 0. The second element should be greater than or equal to the first element, and less than the number of partitions. When using the LAT Client, `partitions_assigned` will be automatically assigned by the client. If it is set in the source configuration provided by the user, it will be overwritten. | Type: | int array | | --------- | --------------------- | | Required: | No | | Default: | Set by the LAT client | `source.compression` The compression method for the files. Currently, only `none` and `gzip` are supported. `none` means that the files are not compressed. Note that this value sets the default compression method for all the files from this source, and it can be overridden by `source_file_groups_compression` per file group. | Type: | string | | --------- | ------ | | Required: | No | | Default: | `none` | `source.chunk_size` The size of a chunk when fetching data. For example, if you have a 40 MB file and the chunk size is 16 MB, you can issue 3 requests sequentially to get the file. The unit is MB. The value must be greater than 0. The chunk size should be larger than or equal to the maximum record size. | Type: | int | | --------- | ---- | | Required: | No | | Default: | `16` | `source.buffer_size` The total buffer size for *all* assigned partitions. This value will be divided by the total number of assigned partitions to calculate each partition’s buffer size. For example, if this value is 4096, `source.partitions` is 16, `source.partitions_assigned` is `[0, 3]`, and you have 2 file groups, then you have a total of (3-0+1)\*2=8 assigned partitions, and each partition will get 4096/8=512 MB as the buffer size. The unit is MB. The value must be greater than 0. The buffer size *per partition* must be at least twice the value of `source.chunk_size`. | Type: | int | | --------- | ------ | | Required: | No | | Default: | `4096` | `source.max_fetch_concurrency` The maximum concurrency when fetching files for each partition. Setting this configuration too high can result in thread contention. The value must be greater than 0. | Type: | int | | --------- | --- | | Required: | No | | Default: | `2` | ### **S3 Source-Specific Configuration** `source.endpoint` The endpoint for the S3 instance, usually starting with `http` or `https`. It can be an IP address or a domain. For example, for AWS S3 in the `us-east-2` region, endpoint would be `https://s3.us-east-2.amazonaws.com`. The S3 Source supports Virtual-hosted-style access and Path-style access. The `s3://` protocol based access is not supported. For more details, see [Methods for Accessing an S3 Bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html) | Type: | string | | --------- | ------ | | Required: | Yes | | Default: | | `source.region` The region for the S3 instance. This is mostly used for AWS S3 instances. | Type: | string | | --------- | ----------- | | Required: | No | | Default: | `us-east-2` | `source.bucket` The bucket from which to get S3 objects. You can override the bucket on a per-file group basis by setting the `source.file_groups..bucket`. This setting is required unless it is specified for every file group. | Type: | string | | --------- | ------ | | Required: | No | | Default: | | `source.path_style_access` Whether to force path style access for the S3 instance. If set to `false`, LAT will try to use the virtual-hosted style (using DNS subdomains), and falls back to the path style access. AWS S3 is deprecating path style access according to [this post](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/). | Type: | boolean | | --------- | ------- | | Required: | No | | Default: | `false` | `source.access_key_id` The access key for the S3 instance. This should be used together with the `source.secret_access_key` setting. If either of them is absent, LAT will default to the next item in the credentials hierarchy. See [S3 Credentials Hierarchy](#s3-credentials-hierarchy). | Type: | string | | --------- | ------ | | Required: | No | | Default: | `null` | `source.secret_access_key` The access secret for the S3 instance. This should be used together with the `source.access_key_id` setting. If either of them is absent, LAT will default to the next item in the credentials hierarchy. See [S3 Credentials Hierarchy](#s3-credentials-hierarchy). | Type: | string | | --------- | ------ | | Required: | No | | Default: | `null` | `source.session_token` Temporary credentials can be made with a combination with the existing secret ID and secret key, and an additional session token. Note that this is only used when both `source.access_key_id` and `source.secret_access_key` are specified. | Type: | string | | --------- | ------ | | Required: | No | | Default: | `null` | `source.retries` Configures the number of retries that will be attempted when download from S3 source fails. | Type: | int | | --------- | -------------------------------------- | | Required: | No | | Default: | Default attempts specified in AWS SDK. | `source.backoff_strategy_base_delay_seconds` Configures the base delay for the backoff strategy of the S3 client in seconds. | Type: | int | | --------- | --- | | Required: | No | | Default: | 1 | `source.backoff_strategy_max_backoff_seconds` Configures the maximum backoff delay for the backoff strategy of the S3 client in seconds. | Type: | int | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Required: | No | | Default: | For the default delay, see the [AWS SDK documentation](https://github.com/aws/aws-sdk-java-v2/blob/master/core/sdk-core/src/main/java/software/amazon/awssdk/core/internal/retry/SdkDefaultRetrySetting.java#L66). | `source.max_pending_connection_acquires` Configures the maximum number of pending acquires allowed by the Netty client. | Type: | int | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Required: | No | | Default: | For the default number of acquires, see the [AWS SDK documentation.](https://github.com/aws/aws-sdk-java-v2/blob/master/http-client-spi/src/main/java/software/amazon/awssdk/http/SdkHttpConfigurationOption.java#L146) | `source.netty_read_timeout_seconds` Configures the read timeout, in seconds, of the Netty client. When you set this value to zero, the system disables the read timeout. The LAT configures read timeouts to be tried again by the S3 Client. | Type: | int | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Required: | No | | Default: | For the default timeout value, see the [AWS SDK documentation.](https://github.com/aws/aws-sdk-java-v2/blob/master/http-client-spi/src/main/java/software/amazon/awssdk/http/SdkHttpConfigurationOption.java#L136) | `source.connection_timeout_seconds` Configures the amount of time in seconds for the Netty client to wait when initially establishing a connection before giving up and timing out. The LAT configures connection timeouts to be retryable by the S3 Client. | Type: | int | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Required: | No | | Default: | For the default timeout value, see the [AWS SDK documentation.](https://github.com/aws/aws-sdk-java-v2/blob/master/http-client-spi/src/main/java/software/amazon/awssdk/http/SdkHttpConfigurationOption.java#L134) | `source.connection_acquisition_timeout_seconds` Configures the amount of time in seconds for the Netty client to wait when acquiring a connection from the pool before giving up and timing out. The LAT configures connection acquisition timeouts to be retryable by the S3 Client. | Type: | int | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Required: | No | | Default: | For the default timeout value, see the [AWS SDK documentation.](https://github.com/aws/aws-sdk-java-v2/blob/master/http-client-spi/src/main/java/software/amazon/awssdk/http/SdkHttpConfigurationOption.java#L137) | `source.requester_pays` Configures whether or not the requester (i.e. the LAT user) should be charged for downloading data from the S3 Requester Pays buckets. This configuration should be set to `true` whenever you request from a Requester Pays bucket, or else the request fails and the bucket owner is charged for the request. | Type: | boolean | | --------- | -------- | | Required: | No | | Default: | `falsed` | ### S3 Credentials Hierarchy The S3 source configuration supports the following hierarchy to obtain S3 credentials. If the LAT does not obtain the credentials at a level, the LAT tries the lower level. * Level 1: Pipeline configuration * Level 2: [AWS SDK Default Credential Provider Chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) * Level 3: (Default) Anonymous access You can choose the level to store the credentials where Level 1 is the highest. Higher levels take precedence for credential storage. ## Load from a Kafka Source The Source allows LAT to connect to a Kafka cluster and process records from one or more topics. You can reference each Kafka topic in the [LAT Transform Configuration](/lat-transform-configuration) and can load into one or more tables. In addition to the LAT specific configuration, this Source supports configuration passthrough to the underlying [Kafka library](https://kafka.apache.org/28/documentation.html#consumerconfigs) which this Source uses. There are some library configurations which are not allowed, and will be noted in this table. ### Example Kafka Source Configuration ```json JSON theme={null} { "source": { "type": "kafka", "kafka": { "bootstrap.servers": "127.0.0.1:9092", "auto.offset.reset": "earliest" } } } ``` ### Kafka Configuration `source.end_offsets_polling_duration` Frequency with which to poll for end offsets for lag calculation, in milliseconds. | Type: | int | | --------- | ----- | | Required: | No | | Default: | 30000 | `source.kafka` Kafka consumer configuration object. See [ConsumerConfig](https://kafka.apache.org/28/documentation.html#consumerconfigs) for details. | Type: | object | | --------- | ------ | | Required: | Yes | | Default: | | `source.kafka` Required Keys: * [`kafka.bootstrap.servers`](https://kafka.apache.org/28/documentation.html#consumerconfigs_bootstrap.servers) * This key is required for connection information to the Kafka servers. Common Keys: * [`kafka.auto.offset.reset`](https://kafka.apache.org/28/documentation.html#consumerconfigs_auto.offset.reset) * This key is commonly set to `earliest` to consume all of the data in an existing Kafka topic. ### Disallowed Kafka Configuration The following Kafka consumer configuration options are not allowed: * `kafka.enable.auto.commit` * LAT will always internally set this configuration to `false`; it is critical for correct operation to do so. * `kafka.key.deserializer` * LAT will always internally configure deserialization * `kafka.value.deserializer` * LAT will always internally configure deserialization If any of the disallowed Kafka library configurations are set, a warning will be logged and their configurations will be ignored. ### Defaulted Kafka Configuration * `kafka.group.id` * If unset. LAT will configure `group.id` internally using the value of `ocient-lat-[$pipeline_id]`. Most users should leave this unset unless they want to explicitly control the `group.id` that is being used. ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Transform Configuration Source: https://docs.ocient.com/lat-transform-configuration Configure transform stages in Ocient LAT pipelines using JMESPath, user-defined transformations, and built-in functions to shape data before it is loaded. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). ## `transform` A transform configuration object. Required keys: * Transform Configuration source only: [transform.topics](#transform-topics) * File sources only: [transform.file\_groups](#transform-topics) ### `transform.topics` \[#transform-topics] A collection of topics and their associated configuration. Keys are topic names and values are [topic configuration objects](#topics). Each `` set as a key in this object represents an topic defined in Kafka. When loading from an `s3` or `local` File Source type, `transform.file_groups` should be used instead of `transform.topics`. Each key in the `file_group` must match a `file_group_name` defined in the Source section of the pipeline configuration. | **Type:** | object | | ------------- | ------ | | **Required:** | Yes | | **Default:** | | ### `transform.database` \[#transform-database] A database to be used to fully qualify any table names that are not fully qualified. For example, if `transform.database` is set to `myDatabase`, a table name of the form `schema.table` will become `myDatabase.schema.table`. | **Type:** | string | | ------------- | ------ | | **Required:** | No | | **Default:** | null | ### `transform.schema` A schema to be used to fully qualify any table names that are not fully qualified. Specifying this property requires that [transform.database](#transform-database) is specified. For example, if `transform.database` is set to `myDatabase` and `transform.schema` is set to `mySchema`: * A table name of the form `table` will become `myDatabase.mySchema.table` * A table name of the form `schema.table` will become `myDatabase.schema.table` * A table name of the form `database.schema.table` will stay as `database.schema.table` | **Type:** | string | | ------------- | ------ | | **Required:** | No | | **Default:** | null | #### Kafka Load Transform Example ```json JSON theme={null} { ... "transform": { "topics": { "topic_1": { "tables": { ... } }, "topic_2": { "tables": { ... } } } } } ``` #### File Based Load Transform Example Unlike Kafka, File loads define file groups in the source section of the pipeline configuration. The "file\_groups" defined in the source and transform sections must match. ```json JSON theme={null} { "source": { "type": "s3", ... "file_groups": { "file_group_1": { ... }, "file_group_2": { ... }, } } "transform": { "file_groups": { "file_group_1": { "tables": { ... } }, "file_group_2": { "tables": { ... } } } } } ``` ### Topics Topic configuration objects. Required keys: * [transform.topics.\.tables ](#transform-topicstopictables) For file based loads, `topics` are replaced by `file_groups`, but all other settings are equivalent. #### `transform.topics..filter` A record filter to apply at the topic level. See [LAT Record Filtering](/lat-record-filtering) for details. | **Type:** | string | | ------------- | ------ | | **Required:** | Yes | | **Default:** | | #### `transform.topics..tables` A collection of tables and their associated configuration. Keys are table names and values are [columns configuration](#columns). | **Type**: | object | | ------------- | ------ | | **Required**: | Yes | | **Default**: | | ### Tables Table configuration objects. Required keys: * [transform.topics.\.tables.\.columns](#transform-topicstopictablestablecolumns) #### `transform.topics..tables.
.filter` A record filter to apply at the table level. See [LAT Record Filtering](/lat-record-filtering) for details. | **Type**: | string | | ------------- | ------ | | **Required**: | Yes | | **Default**: | | #### `transform.topics..tables.
.columns` A collection of columns and their associated configurations. Keys are table names and values are [Columns](#columns). ### Columns Column transformation configurations. Required keys: * [transform.topics.\.tables.\
.columns.\ ](#transform-topicstopictablestablecolumnscolumn) #### `transform.topics..tables.
.columns.` A column transformation keyed by a column name. A column’s value is defined as a transformation expression. The expression will query the record and return a value that is loaded into the associated column. The grammar of these expressions uses [JMESPath](https://jmespath.org/) enhanced with some custom Ocient transformations and User Defined Transformations (UDTs). | **Type**: | string | | ------------- | ------ | | **Required**: | Yes | | **Default**: | | ## Complex Transform Example ```json JSON theme={null} { "transform": { "database": "adtechdb", "schema": "adtech", "topics" : { "dsp.Auctions" : { "tables" : { "auction" : { "columns" : { "auctionid" : "auctionId", "created" : "created", "bidder_nodeid" : "bidder.nodeId", "deals_dealid" : "deals[].dealId" } }, "trafficSource" : { "columns" : { "auctionid" : "auctionId", "created" : "created", "sourcesellerid" : "trafficSource.sourceSellerId", "domainname" : "trafficSource.domainName", "contextualdataset_externalentryid" : "trafficSource.siteContextualProfile.contextualDataSet[].entries[].externalEntryId[]", "dimensions_hw" : "join('x', [dimensions.height, dimensions.width])" } }, "feedback" : { "columns" : { "auctionid" : "auctionId", "created" : "created", "received" : "received", "campaignid" : "EXPLODE(auctionResponseFeedbacks[].campaignId)", "won" : "EXPLODE(auctionResponseFeedbacks[].won)", "winningbid" : "EXPLODE(auctionResponseFeedbacks[].winningBid)", "sourcelossreasoncode" : "EXPLODE(auctionResponseFeedbacks[].sourceLossReasonCode)" } } } } } } } ``` ### Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT Transformation Functions Source: https://docs.ocient.com/lat-transformation-functions Reference for built-in transformation functions in Ocient LAT pipelines, including string, numeric, date, and conditional functions used in transform stages. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). LAT supports many transformation functions to modify data during loading. The functions include standard JMESPath Functions and Ocient-specific transformation functions. Transformation functions in LAT are case sensitive. The standard syntax for transformation function calls is all lower case characters. ## Standard JMESPath Functions The following standard JMESPath Functions are supported in LAT Transformation expressions. For details about using these functions, see [JMESPath Functions](https://jmespath.org/specification.html#functions-expressions). | **JMESPath Function** | **Function Signature** | **Example** | | --------------------- | ------------------------------------------------------------------- | --------------------------------- | | abs | number abs(number \$value) | abs(num\_1) | | avg | number avg(array\[number] \$elements) | avg(array\_1) | | contains | boolean contains(array\|string $subject, any $search) | contains(array\_1, 'test') | | ceil | number ceil(number \$value) | ceil(num\_1) | | ends\_with | boolean ends\_with(string $subject, string $prefix) | ends\_with(str\_1, 'ing') | | floor | number floor(number \$value) | floor(num\_1) | | join | string join(string $glue, array\[string] $stringsarray) | join(',', array\_1) | | keys | array keys(object \$obj) | keys(obj\_1) | | length | number length(string\|array\|object \$subject) | length(array\_1) | | map | array\[any] map(expression→any→any expr, array\[any] elements) | map(\&to\_string(@), array\_1) | | max | number max(array\[number]\|array\[string] \$collection) | max(array\_1) | | max\_by | max\_by(array elements, expression→number\|expression→string expr) | max\_by(array\_1, \&age) | | merge | object merge(\[object \*argument, \[, object \$…]]) | merge(obj\_1, obj\_2, obj\_3) | | min | number min(array\[number]\|array\[string] \$collection) | min(array\_1) | | min\_by | min\_by(array elements, expression→number\|expression→string expr) | min\_by(array\_1, \&age) | | not\_null | any not\_null(\[any $argument \[, any $…]]) | not\_null(val\_1, val\_2, val\_3) | | reverse | array reverse(string\|array \$argument) | reverse(str\_1) | | sort | array sort(array\[number]\|array\[string] \$list) | sort(array\_1) | | sort\_by | sort\_by(array elements, expression→number\|expression→string expr) | sort\_by(array\_1, \&age) | | starts\_with | boolean starts\_with(string $subject, string $prefix) | starts\_with(str\_1, 'chi') | | sum | number sum(array\[number] \$collection) | sum(array\_1) | | to\_array | array to\_array(any \$arg) | to\_array(val\_1) | | to\_string | string to\_string(any \$arg) | to\_string(val\_1) | | type | string type(array\|object\|string\|number\|boolean\|null \$subject) | type(val\_1) | | values | array values(object \$obj) | values(obj\_1) | ## Other Transformation Functions The following transformation functions are also available in LAT transformation expressions. ### `add` Returns the sum of the two arguments. | **Signature:** | `number add(number $a, number $b)` | | -------------- | ------------------------------------------------- | | **Arguments:** | - `a`: first argument
- `b`: second argument | | **Example:** | `add(int_col_0, int_col_1)` | ### `array_cap` Caps the length of an array to a maximum number of elements. | **Signature:** | `array array_cap(Array[any] $array_1, number $max_elements)` | | -------------- | ------------------------------------------------------------------------------------------- | | **Arguments:** | - `array_1`: the array to cap
- `max_elements`: number of elements to cap the array at | | **Example:** | ``array_cap(array_1, `1000`)`` | ### `array_cat` Concatenate two arrays, returning an array. Accepts an array or null value, returning an empty array if both parameters are null. | **Signature:** | `array array_cap(array[any] $array_1, array[any] $array_2)` | | -------------- | ----------------------------------------------------------------------------- | | **Arguments:** | - `array_1`: first array
- `array_2`: second array concatenated to first | | **Example:** | `array_cat(array_1, array_2)` | ### `concat` Concatenates two strings or converts arguments into strings and concatenates them into a string. | **Signature:** | `string concat(any $a, any $b)` | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | - `a`: first argument (automatically converted to string when not specified as a string)
- `b`: second argument (automatically converted to string when not specified as a string) | | **Example:** | ``concat("ocient", `10`)`` | ### `divide` Returns the floating-point division of the two arguments. | **Signature:** | `number divide(number $a, number $b)` | | -------------- | ------------------------------------------------- | | **Arguments:** | - `a`: first argument
- `b`: second argument | | **Example:** | `divide(int_col_0, int_col_1)` | ### `hash_code` Returns an integer hash code for a string, number, or Boolean. | **Signature:** | `number hash_code(string\|number\|boolean $val)` | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | `val`: the value to hash | | **Example:** | `hash_code(null) → null`
`hash_code("Chicago") → -1884315574`
`hash_code("test123") → -1422501792`
``hash_code(`12345`) -> 12345``@@HTML\_TABLE\_TOKEN\_3@@``hash_code(`123.45`) -> -1936584703``@@HTML\_TABLE\_TOKEN\_4@@``hash_code(`true`) -> 3``@@HTML\_TABLE\_TOKEN\_5@@``hash_code(`false`) -> 1`` | ### `if` Returns one of the other arguments based on the truth value of the first. When the first argument is true, return the second argument. When false, return the third. A NULL input into the first argument will be evaluated as false. Both branches are eagerly evaluated. | **Signature:** | `any if(boolean $b, any $if_true, any $if_false)` | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | - `b`: boolean value that determines which of the options to return
- `if_true`: value returned if b is true
- `if_false`: value returned if b is false | | **Example:** | ``if(value > `5`, "North America", "Europe")`` | ### `json` Parses a string into a JSON value for use in further JMESPath operations. All values other than string are returned unmodified. Strings containing null, boolean, number, array, and object types are supported. | **Signature:** | `object json(string $val)` | | -------------- | -------------------------- | | **Arguments:** | `val`: string to parse | | **Example:** | `json("{ \"key\": 10 }")` | ### `lazy_if` Returns one of the other arguments based on the truth value of the first. When the first argument is true, return the second argument. When false, return the third. The second and third arguments are both expressions. Branches are lazily evaluated. | **Signature:** | `any lazy_if(boolean $b, expression $if_true, expression $if_false, any data)` | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | - `b`: boolean value that determines which of the options to return
- `if_true:` expression evaluated and returned if true
- `if_false`: expression evaluated and returned if false
- `data`: information used to evaluate either expression | | **Example:** | `if(value > 5, &"North America", &abs(@), 1.0)` | ### `left` Returns the `num_chars` leftmost characters of the string column. If null, null is returned. If the index is greater than the length of the string, the entire string is returned. | **Signature:** | `string left(string $val, number $num_chars)` | | -------------- | ------------------------------------------------------------------------------------------------- | | **Arguments:** | - `a`: string column
- `num_chars`: number of chars to use from the left of a string column. | | **Example:** | ``left("ocient", `3`)`` | ### `lower_case` Returns lower case version of a string. | **Signature:** | `string lower_case(string $a)` | | -------------- | ------------------------------ | | **Arguments:** | `a`: string to lower case | | **Example:** | `lower_case("OCIENT")` | ### `lpad` Pads an input string to a specified length with a padding string added to the left side. | **Signature:** | `string lpad(string $val, number $length, string $padding)` | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | - `val`: the input string. This cannot be null.
- `length`: the length you want after padding. If the input string is longer than this length, it is truncated to length characters
- `padding`: the padding string. If no padding string is provided, the input string is padded with spaces. | | **Example:** | ``lpad(string_1, `3`, '_')`` | ### `ltrim` Removes leading contiguous instances of a set of characters in a given string. | **Signature:** | `string ltrim(string $val, string $strip_chars)` | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | - `val`: the input string to trim
- `strip_chars`: set of characters that should be removed (default is ' ' if the value is null or not specified) | | **Example:** | `ltrim(null, ' ') → null`
`ltrim(' Chicago ') → 'Chicago '`
`ltrim(' Chicago ', ' ') → 'Chicago '`
`ltrim(' Chicago ', null) → 'Chicago '`
`ltrim('123test123', '321') → 'test123'` | ### `map` An override of JmesPath’s built in map which maps a given expression into elements of an array. This override makes the small adjustment that if the array itself is null, then a null is returned. | **Signature:** | `array map(expression $expr, array[any] array_1)` | | -------------- | ------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | - `expr`: expression the expression to map
- `array_1`: array\[any] the array to which the expression will be mapped | | **Example:** | `map(&if(@ == 1, true, false), [1,2,1,1,2]) → [true, false, true, true, false]` | ### `matches` Returns Boolean for whether this input matches the pattern. Flags can be: * `s` → Pattern.DOTALL * `m` → Pattern.MULTILINE * `i` → Pattern.CASE\_INSENSITIVE * `x` → Pattern.COMMENTS * `q` → Pattern.LITERAL | **Signature:** | `boolean matches(string $val, string $pattern, string $flags)` | | -------------- | ------------------------------------------------------------------------------------------- | | **Arguments:** | - `val`: value to match
- `pattern`: regex to match against
- `args`: regex flags | | **Example:** | `matches(col_1, ".*mytest.*", 'i')` | ### `millis_to_timestamp` Converts an integer number of milliseconds since the Epoch to an Ocient timestamp column. | **Signature:** | `number millis_to_timestamp(number $t)` | | -------------- | -------------------------------------------- | | **Arguments:** | `t`: value to convert to an Ocient timestamp | | **Example:** | `millis_to_timestamp(my_millis)` | ### `multiply` Returns the floating-point multiplication of the two arguments. | **Signature:** | `number multiply(number $a, number $b)` | | -------------- | ------------------------------------------------- | | **Arguments:** | - `a`: first argument
- `b`: second argument | | **Example:** | `multiply(int_col_0, int_col_1)` | ### `nanos_to_timestamp` Converts an integer number of nanoseconds since the Epoch to an Ocient timestamp column. | **Signature:** | `number nanos_to_timestamp(number $t)` | | -------------- | -------------------------------------------------- | | **Arguments:** | **-** `t`: value to convert to an Ocient timestamp | | **Example:** | `nanos_to_timestamp(my_nanosecs)` | ### `normalize_space` Removes leading and trailing whitespace and replaces connected whitespace with a single space. | **Signature:** | `string normalize_space(string $val)` | | -------------- | ------------------------------------- | | **Arguments:** | `val`: string to normalize | | **Example:** | `normalize_space(col_1)` | ### `now, current_timestamp` Returns the current timestamp in nanos. This will correctly load into an Ocient timestamp column | **Signature:** | `number now() \| number current_timestamp()` | | -------------- | -------------------------------------------- | | **Arguments:** | None | | **Example:** | `now(), current_timestamp()` | ### `null_if, nullif` Returns null if the first argument matches any of the following arguments. Can be applied to multiple columns or to compare a column with a constant. Can also be used with more than two arguments in which case returns null if a matches any of the subsequent arguments. Returns the value the first argument if it does not match any other arguments. | **Signature:** | `any null_if(any $val, any $c0, [, any $…])` | | -------------- | --------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | - `val`: value to compare
- `c0`: required value to compare against
- `…` : optional multiple values to compare against | | **Example:** | `null_if(col_a, "NULL", "NONE")` | ### `parse_array` Given a string, parses it into an array of strings. Handles nested arrays as well. | **Signature:** | `array parse_array(string $array_string, string $bracket_char, string $delim_char, int $dimensions)` | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | - `array_string`: the string of the array to be parsed
- `bracket_char`: the opening bracket character to use
- `delim_char`: the delimiter character to use
- `dimensions`: the dimension of this array | | **Example:** | ``parse_array("[ 1, 2, 3]", "[", ",", `1`) ⇒ [1, 2, 3]``@@HTML\_TABLE\_TOKEN\_0@@``parse_array("[[1,2],[3], []]", '[', ',', `2`) ⇒ [[1,2], [3], []]``@@HTML\_TABLE\_TOKEN\_1@@``parse_array("{ \"cat\", \"hat\" }", "{", ",", `1`) ⇒ ["cat", "hat"]`` | ### `record_uuid` Returns string that represents a unique identifier for the record in a given pipeline for a given file\_group or topic. This return value will be consistent over multiple calls in the pipeline's transformation configuration for the same source record. Can be used as a surrogate key when loading multiple tables from the same source record in a pipeline to facilitate joins when no unique identifier exists on the source record. | **Signature:** | `string record_uuid()` | | -------------- | --------------------------------------------------------- | | **Arguments:** | None | | **Example:** | `record_uuid() -> "6064fc40-9961-4f5e-b74d-090458e4a609"` | ### `replace` Returns string with text matching the regex pattern replaced by the replacement text. Flags can be: * `s` → Pattern.DOTALL * `m` → Pattern.MULTILINE * `i` → Pattern.CASE\_INSENSITIVE * `x` → Pattern.COMMENTS * `q` → Pattern.LITERAL | **Signature:** | `string replace(string $val, string $pattern, string $replacement_text, string $flags)` | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `val`: value to search for replacement
- `pattern`: section to replace
- `replacement_text`: replacement string
- `args`: regex flags | | **Example:** | `replace(col_1, '[oO]ld', 'new', 'q')` | ### `right` Returns the `num_chars` rightmost characters of the string column. If null, null is returned. If the index is greater than the length of the string, the entire string is returned. | **Signature:** | `string right(string $a, number $num_chars)` | | -------------- | ------------------------------------------------------------------------------------------------------ | | **Arguments:** | **-** `a`: string column
- `num_chars`: number of chars to use from the right of a string column. | | **Example:** | ``right("ocient", `3`)`` | ### `rpad` Pads an input string to a specified length with a padding string added to the right side. | **Signature:** | `string rpad(string $val, number $length, string $padding)` | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Arguments:** | **-** `val`: the input string. This cannot be null.
- `length`: the length you want after padding. If the input string is longer than this length, it is truncated to length characters
- `padding`: the padding string. If no padding string is provided, the input string is padded with spaces. | | **Example:** | ``rpad(string_1, `3`, '_')`` | ### `rtrim` Removes trailing contiguous instances of a set of characters in a given string. | **Signature:** | `string rtrim(string $val, string $strip_chars)` | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `val`: the input string to trim
- `strip_chars`: set of characters that should be removed (default is ' ' if the value is null or not specified) | | **Example:** | `rtrim(null, ' ') → null`
`rtrim(' Chicago ') → ' Chicago'`
`rtrim(' Chicago ', ' ') → ' Chicago'`
`rtrim(' Chicago ', null) → ' Chicago'`
`rtrim('123test123', '321') → '123test'` | ### `sample` Returns true or false based on the given sample rate. The sample rate must be a number. Commonly used inside of `IF` statements. An exception will be thrown if the sample rate is outside of \[0.0, 1.0]. | **Signature:** | `boolean sample(number $rate)` | | -------------- | --------------------------------------------------------- | | **Arguments:** | `rate`: rate to return true. Must be between \[0.0, 1.0]. | | **Example:** | ``sample(`0.5`)`` | ### `st_forcepolygonccw` Forces a polygon to have a counterclockwise rotation of the outer polygon ring and a clockwise rotation of the inner polygon ring. You can use this function to convert polygons extracted from systems that interpret polygon rotation differently. | **Signature:** | `geometry st_forcepolygonccw(string $wkt_geometry)` | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Arguments:** | `wkt_geometry`: WKT string representing the polygon to rotate. Also accepts WKT of POINT or LINESTRING and returns those types unmodified. | | **Example:** | `st_forcepolygonccw('POLYGON(…)')` | ### `st_geomfromewkb` Returns a geometry from an Extended Well-Known Binary (EWKB) and Well-Known Binary (WKB) representations of a geometry. You can load the result of this function into POINT, LINESTRING, and POLYGON column types. | **Signature:** | `geometry st_geomfromewkb(string ewkb)` | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | `wkb`: hex encoded string Extended WKB or WKB representation of a geometry. EWKB or WKB can be standalone hexadecimal characters or can be prefixed with either 0x or \x. | | **Example:** | `st_geomfromewkb('0103…')`
`st_geomfromewkb('0x0103…')`
`st_geomfromewkb('\x0103…')` | ### `st_point` Constructs an POINT from numeric coordinates that represent the longitude and latitude of the point. | **Signature:** | `geometry st_point(float longitude, float latitude)` | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Arguments:** | **-** `longitude`: floating point longitude of the point to construct
- `latitude`: floating point latitude of the point to construct | | **Example:** | `st_point(float_col_1, float_col_2)` | ### `substring` Returns the substring beginning at `start_index` (with indexes beginning at 1) to the end of the string, or to `num_chars` characters. | **Signature:** | `string substring(string $val, number $start_index, [number $num_chars])` | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | `val`: string to substring
`start_index`: index to start at. First character is index 1.
`num_chars`: optional. The maximum number of chars to return | | **Example:** | ``substring(str_col, `10`)``@@HTML\_TABLE\_TOKEN\_0@@``substring(str_col, `10`, `5`)``@@HTML\_TABLE\_TOKEN\_1@@``substring('Chicago Bears', `3`) → 'icago Bears'``@@HTML\_TABLE\_TOKEN\_2@@``substring('Chicago Bears', `3`, `4`) → 'icag'`` | ### `substring_after` Returns the string after the first occurrence of the needle. | **Signature:** | `string substring_after(string $val, string $needle)` | | -------------- | --------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `val`: string to substring
- `needle`: delimiter to use to designate where to return subsequent substring. | | **Example:** | `substring_after('my_text', '_') → 'text'` | ### `substring_before` Returns the string before the first occurrence of the needle. Both parameters are literal strings. | **Signature:** | `string substring_before(string $val, string $needle)` | | -------------- | ---------------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `val`: string to substring
- `needle`: delimiter to use to designate where to return prior substring. | | **Example:** | `substring_before('my_text', '_') → 'my'` | ### `subtract` Returns the difference of the two arguments. | **Signature:** | `number subtract(number $a, number $b)` | | -------------- | ----------------------------------------------------- | | **Arguments:** | **-** `a`: first argument
- `b`: second argument | | **Example:** | `subtract(int_col_0, int_col_1)` | ### `to_array_length` Turns any JSON object into an array with N copies of that object. | **Signature:** | `array[any] to_array_length(any $val, number $n)` | | -------------- | ----------------------------------------------------------------------------- | | **Arguments:** | **-** `val`: value to copy into an array
- `n`: number of copies to make | | **Example:** | ``to_array_length(value_col, `3`)`` | ### `to_binary` Converts a string to binary data for loading into an Ocient binary or hash column. If `mode` is `'hex'`, `data` is parsed as a sequence of hexadecimal digits. Note that the sequence of digits must not begin with `0x`. Otherwise, `mode` must be the name of a character encodings listed [here](https://docs.oracle.com/en/java/javase/17/intl/supported-encodings.html), and `data` is encoded in that encoding. | **Signature:** | `binary to_binary(string $data, string $mode)` | | -------------- | ----------------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `data`: data to convert to binary
- `mode`: either 'hex' or the name of a standard character encoding | | **Example:** | `to_binary(str_col, 'utf-16')` | ### `to_date` Converts a string to a date for loading into an Ocient date column. Format string follows these [formatting rules](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/time/format/DateTimeFormatter.html). | **Signature:** | `number to_date(string $date, string $format_string)` | | -------------- | ------------------------------------------------------------------------------------------------ | | **Arguments:** | **-** `date`: date string to convert
- `format_string`: format string to use for conversion | | **Example:** | `to_date(str_col, 'yyyy-MM-dd')` | ### `to_time` Converts a string into a time of day for loading into an Ocient time column. Format string follows these [formatting rules](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/time/format/DateTimeFormatter.html). | **Signature:** | `number to_time(string $time, string $format_string)` | | -------------- | ------------------------------------------------------------------------------------------------ | | **Arguments:** | **-** `time`: time string to convert
- `format_string`: format string to use for conversion | | **Example:** | `to_time(str_col, 'HH:mm:ss.nnnnnnnnn')` | ### `to_timestamp` Converts a string to timestamp for loading into an Ocient timestamp column. Format string follows these [formatting rules](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/time/format/DateTimeFormatter.html). | **Signature:** | `number to_timestamp(string $timestamp, string $format_string)` | | -------------- | ---------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `timestamp`: timestamp string to convert
- `format_string`: format string to use for conversion | | **Example:** | `to_timestamp(str_col, 'yyyy-MM-dd HH:mm:ss.SSS')` | ### `to_tuple` Works on tuples that get tokenized at the first nesting level by a given delimiter. Nested tuples, arrays or objects are considered entirely. For each tokenized tuple element, a transformation function can be specified that is applied to the tuple element. Tokenized elements are always treated as strings, requiring additional transformation when being treated as numbers. | **Signature:** | `array to_tuple(string $tuple_string, string $bracket_char, string $delim_char [, expression $transformation_function …])` | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `tuple_string`: the string of the tuple to be parsed.
Must be encapsulated in the given bracket character and its closing peer. Empty tuple values are treated as null values.
If the whole argument is NULL, the function returns NULL ignoring all given expressions in transformation\_function…
- `bracket_char`: the bracket character to use for parsing and result generation
- `delim_char`: the delimiter character to use for parsing and result generation.

The supported delimiters are: '(', '\[' and '\{'

- `avg(…)`
- `keys(…)`
- `map(…)`
- `sum(…)`
- `value(…)`
- `type(…) - partially for boolean and object types` | | **Example:** | - `to_tuple((-1,2), '(', ',', &abs(to_number(@)), &@) → '(1.0,2)'`
- ``to_tuple((,'8',2), '(', ',', &if(@ != null, @, 'missing'), &if(length(@) > `0`, @, 'missing'), &to_number(@) ) → '("missing","'8'",2)'`` | ### `tokenize` Returns array of strings after splitting on the regex pattern. Flags can be: * `s` → Pattern.DOTALL * `m` → Pattern.MULTILINE * `i` → Pattern.CASE\_INSENSITIVE * `x` → Pattern.COMMENTS * `q` → Pattern.LITERAL | **Signature:** | `array[string] tokenize(string $val, string $pattern, string $flags)` | | -------------- | ------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `val`: string to tokenize
- `pattern`: token regex pattern
- `flags`: regex flags | | **Example:** | `tokenize(col_1, ''), 'i`') | ### `translate` Replaces characters according to a map of characters. If the to\_chars string is shorter than the from\_chars string, the function removes the characters that are not mapped. | **Signature:** | `string translate(string $val, string $from_chars, string $to_chars)` | | -------------- | ----------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `val`: string to translate
- `from_char`: from character map
- `to_chars`: to character map | | **Example:** | `translate('Lorem Ipsum', 'mnop', '1234') → 'L3re1 I4su1'` | ### `trim` Removes leading and trailing contiguous instances of a set of characters in a given string. | **Signature:** | `string trim(string $val, string $strip_chars)` | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `val`: the input string to trim
- `strip_chars`: set of characters that should be removed (default is ' ' if the value is null or not specified) | | **Example:** | `trim(null, ' ') → null`
`trim(' Chicago ') → 'Chicago'`
`trim(' Chicago ', ' ') → 'Chicago'`
`trim(' Chicago ', null) → 'Chicago'`
`trim('123test123', '321') → 'test'` | ### `truncate` Truncates a decimal number to have a specified number of digits after its decimal point. If `decimal` is null, this function returns null. During the transformation stage, the LAT stores numbers as integers or floating point numbers by default. However, floating point numbers are inexact. For this reason, this function might return unexpected results. For example, ``truncate(`0.29`, `2`)`` evaluates to `0.28`, not `0.29` (because `0.29` is really `0.28999999...`). | **Signature:** | `number truncate(number $decimal, number $decimal_places)` | | -------------- | --------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `decimal`: the number to truncate
- `decimal_places`: the number of digits to keep after the decimal point | | **Example:** | ``truncate(decimal_col, `2`)`` | ### `tuple_element_transformation` Works on an element of a given array and applies the specified expression to that element. The function returns an array or null if specified. | **Signature:** | `array tuple_element_transformation(array $array_values, number $array_index, expression $transformation_function)` | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `array_values`: an array of values.
If the whole argument is NULL, the function returns NULL.
- `array_index`: a number pointing to an element in the array. The first index of the array is 0
- `transformation_function`: a transformation functions to apply to array element specified by array\_index. | | **Example:** | `tuple_element_transformation([-1.0,2,3, '0', abs(@)) → [1.0,2,3] ]` | ### `unique` Return all unique values in an array. | **Signature:** | `array[any] unique(array[any] $vals)` | | -------------- | -------------------------------------------------------------------- | | **Arguments:** | `vals`: array to return unique values from, removing all duplicates. | | **Example:** | `unique(vals_col)` | ### `upper_case` Returns upper case version of string. | **Signature:** | `string upper_case(string $a)` | | -------------- | ------------------------------ | | **Arguments:** | `a`: string to upper case | | **Example:** | `upper_case("ocient")` | ### `width_bucket` Returns the bucket number of the queried value in a histogram starting at min, ending at max, and consisting of num\_buckets count of buckets. * The values are inclusive on the lower bound and exclusive of the upper bound so that all buckets are the same width. * Also note that requesting 100 buckets will actually get you 102 possible buckets. Bucket 0 through 101. Where 0 captures all values below the minimum and 101 captures all values at the maximum and beyond. | **Signature:** | `number width_bucket(number $val, number $min, number max, number $num_buckets)` | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Arguments:** | **-** `val`: value to determine the bucket number for
- `min`: min value of histogram
- `max`: max value of histogram
- `num_buckets`: number of buckets in histogram | | **Example:** | ``width_bucket(my_col, `0`, `100`, `20`)`` | ### `zip` Zips N one-dimensional arrays into a single two-dimensional array with each inner array containing N elements. The resulting outer array will have a length equal to the max length of any input array. For input arrays smaller than the max length, null elements will be placed in the inner arrays. | **Signature:** | `array[any] zip(array[any] $a0, array[any] $a1, [, array[any] $…])` | | -------------- | ------------------------------------------------------------------------------------------------- | | **Arguments:** | **-** `a0`: array to zip
- `a1`: array to zip
- `…` : Optional additional arrays to zip | | **Example:** | `zip(array_col0, array_col1, array_col2)` | ## Special Functions ### EXPLODE Performs an explosion on all indicated columns, producing one row per value in the exploded array. This can be applied to multiple columns in the same table. Non-exploded columns are held constant and exploded columns are zipped together in array order. If multiple exploded columns have different array lengths, the record is exploded into N records where N is the max length. Columns with array lengths less than N will have `null` values after each of their values are exploded. You can explode a maximum of 8,192 array elements for each source record. ### Usage `EXPLODE(array[any] arr)` can be used like any other JMESPath function on a JSON Array type with a few limitations: * `EXPLODE` can only be used as the outermost function in a JMESPath expression * `EXPLODE` only works on a single dimension. Multidimensional explode is not supported. ### Examples **Single-Column EXPLODE** Transform Configuration: ```json JSON theme={null} { "transform": { "topics" : { "topic0" : { "tables" : { "table0" : { "columns" { "col0": "EXPLODE(array0)", "col1": "integer0" } } } } } } } ``` Input: ```json JSON theme={null} { "array0": [0, 1, 2, 3], "integer0": 4 } ``` Output: ```json JSON theme={null} [ { "col0": 0, "col1": 4 }, { "col0": 1, "col1": 4 }, { "col0": 2, "col1": 4 }, { "col0": 3, "col1": 4 } ] ``` **Multi-Column EXPLODE** Transform Configuration: ```json JSON theme={null} { "transform": { "topics" : { "topic0" : { "tables" : { "table0" : { "columns" : { "col0": "EXPLODE(array0)", "col1": "EXPLODE(array1)", "col2": "integer0" } } } } } } } ``` Input: ```json JSON theme={null} { "array0": [0, 1, 2, 3], "array1": [0, 1, 2], "integer0": 4 } ``` Output: ```json JSON theme={null} [ { "col0": 0, "col1": 0, "col2": 4 }, { "col0": 1, "col1": 1, "col2": 4 }, { "col0": 2, "col1": 2, "col2": 4 }, { "col0": 3, "col1": null, "col2": 4 } ] ``` Note that in the 4th record `col1` has a `null` value because `array1` only has 3 values while `array0` has 4. ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # LAT User-Defined Transformations Source: https://docs.ocient.com/lat-user-defined-transformations Define custom transformations in Ocient LAT pipelines using Groovy or Java to apply project-specific business logic when reshaping incoming data records. Data Pipelines are now the preferred method for loading data into the System. For details, see [Load Data](/load-data). In addition to built-in transformations, users can extend LAT by writing custom user-defined transformations (UDTs). To bootstrap UDT development, contact Ocient Support. ## UDT Interface A UDT is implemented as a [JMESPath Java Function](https://github.com/burtcorp/jmespath-java/blob/master/jmespath-core/src/main/java/io/burt/jmespath/function/Function.java). A Function is comprised of three elements: 1. **name** - the function’s name determines how it can be called in a user pipeline e.g., `negate` 2. **arguments** - what inputs the function expects to be called with 3. **body** - the function’s implementation, including return value UDT names must be globally unique. LAT validates UDT names on startup to ensure that no two UDT names collide with each other and that no UDT name collides with an LAT builtin transformation. This example function demonstrates these concepts: ```java Java theme={null} // negate(number) - takes an input number and changes its sign // function name - derived from the class name, NegateFunction -> negate public class NegateFunction extends BaseFunction { public NegateFunction() { super( // function arguments; this function takes exactly one argument of type number ArgumentConstraints.typeOf(JmesPathType.NUMBER)); } // function body protected T callFunction(Adapter runtime, List> arguments) { double original = runtime.toNumber(arguments.get(0).value()).doubleValue(); return runtime.createNumber(original * -1); } } ``` ## UDT Packaging One or more UDTs can be packaged in a jar file for consumption by LAT. In addition to the UDT implementations themselves, the jar file must contain a services file containing a mapping of all the UDTs the jar should expose for use by LAT. This service file must be located at `META-INF/services/io.burt.jmespath.function.Function` in the resulting jar, and its contents should be as follows: ```Text Text theme={null} com.ocient.myudt.NegateFunction ``` The file at the path `META-INF/services/io.burt.jmespath.function.Function` should include the fully qualified name of one or more UDTs contained within this jar to be used by LAT. Each UDT should be specified on its own line in the file. ## Dependencies If the UDT has dependencies on third party libraries, those libraries must be packaged in the same jar file. This is commonly referred to as a "fat jar", "uber jar", or "jar with dependencies." Instructions for building such a jar file with Maven™ can be found [here](https://maven.apache.org/plugins/maven-assembly-plugin/single-mojo.html). ## UDT Deployment LAT can be configured to search for UDT jars in a particular directory. See [LAT Source Configuration](/lat-source-configuration) for details. LAT can read multiple UDTs across multiple jar files. On startup, LAT will read all available UDTs into its configuration. To update the set of UDTs available to LAT, such as to add an additional UDT jar, a process restart is required. ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) # Linestring Constructors Source: https://docs.ocient.com/linestring-constructors Construct linestring geometries in OcientGeo using ST_LINESTRING and related functions to build spatial line objects from point coordinates and arrays. LINESTRING constructors use geospatial data to create a LINESTRING object. ## ST\_LINEFROMTEXT Alias for ST\_LINESTRING(CHAR). Creates a `LINESTRING` from a specified `CHAR`. The `CHAR` must be a `LINESTRING` value in WKT format. **Syntax** ```sql SQL theme={null} ST_LINEFROMTEXT(char) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | -------------------------------------------------------------------- | | `char` | `CHAR` | A character value in WKT format to be used to create a `LINESTRING`. | **Example** ```sql SQL theme={null} SELECT ST_LINEFROMTEXT('LINESTRING(1 2)'); ``` *Output*: `LINESTRING(1.000000 2.000000)` ## ST\_LINEFROMWKB Alias for ST\_LINESTRING(BINARY). Creates a `LINESTRING` from the specified `BINARY`. The `BINARY` value must be a `LINESTRING` in WKB format. **Syntax** ```sql SQL theme={null} ST_LINEFROMWKB(binary) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ----------------------------------------------------------------- | | `binary` | `BINARY` | A binary value in WKB format to be used to create a `LINESTRING`. | **Example** ```sql SQL theme={null} SELECT ST_LINEFROMWKB( BINARY( '0x010200000002000000000000000000f03f000000000000004000000000000008400000000000001040' ) ); ``` *Output*: `LINESTRING(1 2,3 4)` ## ST\_LINEFROMEWKT Creates a `LINESTRING` from the specified `CHAR`. The `CHAR` value must be a `LINESTRING` in EWKT format. `ST_LINEFROMEWKT` is functionally the same as `ST_LINESTRING(CHAR)`. The input string must include an `SRID=…​;` value. However, the database ignores this component as all geography types are SRID 4326. **Syntax** ```sql SQL theme={null} ST_LINEFROMEWKT('SRID=value;char') ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `char` | `CHAR` | A `LINESTRING` value in WKT format to be used to create a `LINESTRING`.
To function properly, you must prefix this argument with an SRID value in the same string. | | `value` | `INTEGER` | Any integer value. This is required, however, the database ignores this component and defaults to SRID 4326. | **Example** ```sql SQL theme={null} SELECT ST_LINEFROMEWKT('SRID=4326;LINESTRING(1 2)'); ``` *Output*: `LINESTRING(1.000000 2.000000)` ## ST\_LINEFROMGEOJSON Creates a `LINESTRING` represented by the specified GeoJSON. The specified GeoJSON can represent a GeoJSON `POINT` or `LINESTRING` type. Valid GeoJSON formats follow [ standards](https://datatracker.ietf.org/doc/rfc7946/), and you can generate them by using [ST\_ASGEOJSON](/conversion-functions#st_asgeojson). If you specify an invalid GeoJSON, the behavior of the function is undefined. **Syntax** ```sql SQL theme={null} ST_LINEFROMGEOJSON(geojson [, geodesic ]) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `geojson` | `CHAR` | A GeoJSON value that represents a `POINT` or `LINESTRING` type that is used to create a `LINESTRING`.
If `geojson` is NULL, the function returns NULL. | | `geodesic` | `BOOLEAN` | Optional. A `BOOLEAN` value that determines if the specified `geojson` should be converted from a planar to a geodesic representation.
If you specify `geodesic` as `TRUE`, the function adds points to the resulting `LINESTRING` such that it remains within 10 meters of the original planar line.
If unspecified, `geodesic` defaults to `FALSE`.
If `geodesic` is NULL, the function returns NULL. | **Examples** In this example, the function converts an empty GeoJSON value of a `LINESTRING`. ```sql SQL theme={null} SELECT ST_LINEFROMGEOJSON('{"type":"LineString","coordinates":[]}',false); ``` *Output*: `LINESTRING EMPTY` In this example, the function converts a GeoJSON value of a `LINESTRING`. ```sql SQL theme={null} SELECT ST_LINEFROMGEOJSON( '{"type":"LineString","coordinates":[[1,1],[1,5],[5,5],[5,1]]}', false); ``` *Output*: `LINESTRING(1 1, 1 5, 5 5, 5 1)` ## ST\_LINESTRING Alias for ST\_MAKELINE. Creates a `LINESTRING` based on the specified inputs. ### ST\_LINESTRING(geoArray) \[#st\_linestring-geoarray] Creates a `LINESTRING` from the specified array of either `LINESTRING` or `POINT` values. **Syntax** ```sql SQL theme={null} ST_LINESTRING(geo_array) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | -------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | `geo_array` | `ARRAY` of geographies, either `LINESTRING` or `POINT` values. | An array of `LINESTRING` or `POINT` values to be used to create a unified `LINESTRING`. | **Example** ```sql SQL theme={null} SELECT ST_LINESTRING( ST_LINESTRING [](ST_LINESTRING(ST_POINT(1, 2), ST_POINT(3, 4)))); ``` *Output*: `LINESTRING(1.000000 2.000000, 3.000000 4.000000)` ### ST\_LINESTRING(binary) Creates a `LINESTRING` from the specified `BINARY`. The `BINARY` value must be a `LINESTRING` in WKB format. **Syntax** ```sql SQL theme={null} ST_LINESTRING(binary) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | -------------------------------------------------------------------------------------------------------------- | | `binary` | `BINARY` | A `BINARY` value to be used to create a `LINESTRING`. The `BINARY` value must be a `LINESTRING` in WKB format. | **Example** ```sql SQL theme={null} SELECT ST_LINESTRING(BINARY('0x010200000002000000000000000000f03f000000000000004000000000000008400000000000001040')); ``` *Output*: `LINESTRING(1 2,3 4)` ### ST\_LINESTRING(char) \[#st\_linestring-char] Creates a `LINESTRING` from the specified `CHAR`. The `CHAR` value must be a `LINESTRING` in WKT format. **Syntax** ```sql SQL theme={null} ST_LINESTRING(char) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------ | | `char` | `CHAR` | A `CHAR` value to be used to create a `LINESTRING`. The argument must be a `LINESTRING` in WKT format. | **Example** ```sql SQL theme={null} SELECT ST_LINESTRING('LINESTRING(1 2)'); ``` *Output*: `LINESTRING(1.000000 2.000000)` ### ST\_LINESTRING(geo1, geo2) \[#st\_linestring-geo1-geo2] Creates a `LINESTRING` consisting of two separate geographic arguments, both of which must be either a `POINT` or `LINESTRING`. You can mix geographic types, meaning one argument can be a `POINT` while the other is a `LINESTRING`. **Syntax** ```sql SQL theme={null} ST_LINESTRING(geo1, geo2) ``` | **Argument** | **Data** **Type** | **Description** | | -------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `geo1`, `geo2` | `POINT` or `LINESTRING` | Two separate geographic arguments to be consolidated into a single `LINESTRING`.

Inputs can be the same type or one of each. | **Example** ```sql SQL theme={null} SELECT ST_LINESTRING(ST_POINT(1, 2), ST_POINT(1,3)); ``` *Output*: `LINESTRING(1.000000 2.000000, 1.000000 3.000000)` ## ST\_MAKELINE Alias for [ST\_LINESTRING](#st_linestring). ### ST\_MAKELINE(geo1, geo2) Alias for [ST\_LINESTRING(geo1, geo2)](#st_linestring-geo1-geo2). ### ST\_MAKELINE(CHAR) Alias for [ST\_LINESTRING(char)](#st_linestring-char). ### ST\_MAKELINE(geoArray) Alias for [ST\_LINESTRING(geoArray)](#st_linestring-geoarray). ## Related Links [Geospatial Data Types](/data-types#geospatial-data-types) [Linestring Functions](/linestring-functions) [Conversion Functions](/conversion-functions) # Linestring Functions Source: https://docs.ocient.com/linestring-functions Reference for OcientGeo linestring functions to measure, transform, and inspect linear spatial geometries including length, projection, and conversion. LINESTRING functions can perform alterations or access descriptive information on LINESTRING objects. ## ST\_ADDPOINT Adds a `POINT` to the given `LINESTRING` at the specified 0-indexed location. **Syntax** ```sql SQL theme={null} ST_ADDPOINT(geo_linestring, geo_point_to_add [, location ] ) ``` | **Argument** | **Data** **Type** | **Description** | | ------------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `geo_linestring` | `LINESTRING` | A linestring to be altered by adding the `geo_point_to_add` at a specified location.
The function throws an error if `geo_linestring` is any data type except for `LINESTRING` or NULL. | | `geo_point_to_add` | `POINT` | A point value to be added to the `geo_linestring`. | | `location` | `INTEGER` | Optional. Represents a numeric 0-indexed location in the `geo_linestring` where `geo_point_to_add` is appended.
If you do not specify this value, the function defaults to appending the `geo_point_to_add` argument to the end of `geo_linestring`. A location value of -1 also appends to the end.
The function throws an error if the `location` argument is outside the bounds of the `geo_linestring` or if it is a value of less than -1. | **Example** ```sql SQL theme={null} SELECT ST_ADDPOINT( ST_LINESTRING('LINESTRING(1 2, 1 3)'), ST_POINT(1, 4), 1); ``` *Output*: `LINESTRING(1.000000 2.000000, 1.000000 4.000000, 1.000000 3.000000)` ## ST\_ENDPOINT Returns the endpoint of a specified `LINESTRING`. The returned value is a `POINT`. **Syntax** ```sql SQL theme={null} ST_ENDPOINT(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `geo` | `LINESTRING` | A line to be computed to return its endpoint.
If this value is a `POLYGON`, `POINT`, or is empty, then the function returns NULL.
If this value is a non-geospatial type, the function throws an error. | **Example** ```sql SQL theme={null} SELECT ST_ENDPOINT(ST_LINESTRING('LINESTRING(1 2, 1 3, 1 4)')); ``` *Output*: `POINT(1,4)` ## ST\_LINEINTERPOLATEPOINT Returns a `POINT` along a `LINESTRING` based on a specified fraction of its total length. **Syntax** ```sql SQL theme={null} ST_LINEINTERPOLATEPOINT(line, fraction) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `line` | `LINESTRING` | A line used to compute an interpolated point. | | `fraction` | `DOUBLE` | A decimal value between `0` and `1` that represents a fraction of the total length of the `line`. The function returns a point at this fraction of the length.
The function throws an error if this value is not between `0` and `1`. | **Example** In this example, the function computes a point halfway (0.5) along the specified line. ```sql SQL theme={null} SELECT ST_LINEINTERPOLATEPOINT(ST_LINESTRING('LINESTRING(1 2, 2 1)'), 0.5); ``` *Output*: `POINT(1.5 1.5)` ## ST\_LINELOCATEPOINT Similar to [ST\_LINEINTERPOLATEPOINT](#st_lineinterpolatepoint), this function computes a fraction based on where a specified `POINT` is located along the length of a specified `LINESTRING`. **Syntax** ```sql SQL theme={null} ST_LINELOCATEPOINT(line, point) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `line` | `LINESTRING` | A `LINESTRING` used to compute a fraction based on the interpolated `point` value. | | `point` | `POINT` | A `POINT` used to calculate a fraction based on where it is located along the `line` value.
If the function cannot interpolate this value from the `line`, the function returns `0.0`. | **Example** In this example, the function computes a fraction that represents where the point (1.5, 1.5) lies along the line. ```sql SQL theme={null} SELECT ST_LINELOCATEPOINT( ST_LINESTRING('LINESTRING(1 2, 2 1)'), ST_POINT(1.5, 1.5)); ``` *Output*: `0.5` ## ST\_LINESUBSTRING Returns a `LINESTRING` that is a substring of a specified line that starts and ends at the specified fractions of its total length. **Syntax** ```sql SQL theme={null} ST_LINESUBSTRING(line, start_fraction, end_fraction) ``` | **Argument** | **Data** **Type** | **Description** | | ---------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `line` | `LINESTRING` | A `LINESTRING` used to compute a substring line based on the `start_fraction` and `end_fraction` values. | | `start_fraction` | `DOUBLE` | A decimal value between `0` and `1` that represents a fractional point of the total length of the `line`. The function uses this fractional point as the start point for the substring.
The function throws an error if the `start_fraction` is greater than the `end_fraction` | | `end_fraction` | `DOUBLE` | A decimal value between `0` and `1` that represents a fractional point of the total length of `line`. The function uses this fractional point as the endpoint for the substring. | **Example** In this example, the function generates a subsection of the provided linestring, spanning from the `start_fraction` value (0.5) that represents the halfway point of the original linestring, to the `end_fraction` value (1.0) that represents the endpoint. ```sql SQL theme={null} SELECT ST_LINESUBSTRING(ST_LINESTRING('LINESTRING(1 2, 2 1)'), 0.5, 1.0); ``` *Output*: `LINESTRING(1.50 1.50, 2.0 1.0)` ## ST\_POINTN Returns the `POINT` value at a specified index of the specified `LINESTRING`. If out of bounds, the function returns NULL. **Syntax** ```sql SQL theme={null} ST_POINTN(geo, index) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `geo` | `LINESTRING` | A `LINESTRING` that is used to return one of its point values.
If you specify a `POINT` or `POLYGON` value, this function returns NULL. | | `index` | `INTEGER` | An index location of one of the point values in the `geo` value.
This is one-based, meaning the first index location starts at `1`.
Negative values count backward from the end of the linestring. In other words, an index value of `-1` returns the last point value. | **Example** ```sql SQL theme={null} SELECT ST_POINTN( ST_LINESTRING( ST_POINT [](ST_POINT(1, 2), ST_POINT(1, 3), ST_POINT(1, 4))), 3); ``` *Output*: `ST_POINT(1,4)` ## ST\_REMOVEPOINT Removes a `POINT` value at a specified index from the specified line. **Syntax** ```sql SQL theme={null} ST_REMOVEPOINT(geo, index) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `geo` | `LINESTRING` | A `LINESTRING` that is to be altered by removing one of its point values. | | `index` | `INTEGER` | An index of one of the point values in the `geo` value.
This is zero-based, meaning the first index location starts at `0`, and the last index is `length(geo)-1`. | **Example** ```sql SQL theme={null} SELECT ST_REMOVEPOINT(ST_LINESTRING('LINESTRING(1 2, 1 3, 1 4)'), 1); ``` *Output*: `LINESTRING(1.0 2.0, 1.0 4.0)` ## ST\_SETPOINT Replaces a `POINT` value in a specified `LINESTRING` at a specified index. The function returns the altered `LINESTRING` with the replaced point. **Syntax** ```sql SQL theme={null} ST_SETPOINT(geo, index, geo_point_to_replace) ``` | **Argument** | **Data** **Type** | **Description** | | ---------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `geo` | `LINESTRING` | A `LINESTRING` that is to be altered with a new point value. | | `index` | `INTEGER` | An index location of one of the point values in the `geo` linestring.
This is zero-based, meaning the first index location starts at `0`.
Negative values count backward from the end of the `geo` value. In other words, an index value of `-1` returns the last point value. | | `geo_point_to_replace` | `POINT` | A `POINT` value to replace the value at the specified `index` location.
If this value is not a `POINT` type or NULL, the function throws an error. | **Example** In this example, the function replaces the point value located at the `-2` index location, meaning it is the second from the last in the sequence. ```sql SQL theme={null} SELECT ST_SETPOINT( ST_LINESTRING('LINESTRING(1 2, 1 3, 1 4)'), -2, ST_POINT(1, 5)); ``` *Output*: `LINESTRING(1.000000 2.000000, 1.000000 5.000000, 1.000000 4.000000)` ## ST\_STARTPOINT Returns the starting `POINT` value of the line. If this value is empty, the function returns NULL. If you specify a `POLYGON`, `POINT`, or NULL, then the function returns NULL. If you specify a non-geography value, the function throws an error. **Syntax** ```sql SQL theme={null} ST_STARTPOINT(geo) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `geo` | `LINESTRING` | A line to be computed to return its starting point.
If this value is a `POLYGON`, `POINT`, or is empty, then the function returns NULL.
If this value is a non-geospatial type, the function throws an error. | **Example** ```sql SQL theme={null} SELECT ST_STARTPOINT(ST_LINESTRING('LINESTRING(1 2, 1 3, 1 4)')); ``` *Output*: `ST_POINT(1,2)` ## Related Links [Geospatial Data Types](/data-types#geospatial-data-types) [Linestring Constructors](/linestring-constructors) # Load and Analyze Data Source: https://docs.ocient.com/load-and-analyze-data End-to-end tutorial: connect to Ocient, create a database and table, load sample data with a data pipeline, validate the results, and run analytic queries. This tutorial details the core workflow from start to finish. You connect to the system, create a database and table, load sample data from a source, validate the loaded data, and execute analytic queries. The example uses a sample data set that covers every supported Ocient data type. By the end of this process, you have a working table with loaded data that you can query immediately. ## Prerequisites The tutorial requires: * Network access to the SQL Nodes in your Ocient System and the IP address or hostname of at least one SQL Node. * A valid username and password with permissions to create databases, tables, and data pipelines. * You have installed the Ocient JDBC driver and CLI on your local machine. For setup instructions, see [Connect Using JDBC](/connect-using-jdbc). If you do not have the JDBC driver installed, you can also connect using `pyocient`, the Ocient driver. For setup instructions, see [Connect Using pyocient](/connect-using-pyocient). ## Step 1: Connect to the Ocient System Start the JDBC CLI and connect to a SQL Node. Replace the hostname `` and port `` placeholders with the values for your system. The default SQL Node port is `4050`. ```sql SQL theme={null} CONNECT TO jdbc:ocient://:/system; ``` After a successful connection, you see the Ocient CLI prompt. ```shell Shell theme={null} Ocient> _ ``` For details on connection methods and troubleshooting, see [Connect to Ocient](/connect-to-ocient). ## Step 2: Create a Database and Schema Create a database to hold your sample data. Using a dedicated database keeps your work isolated from other applications on the same system. ```sql SQL theme={null} CREATE DATABASE test; ``` Switch your connection to the new database. ```sql SQL theme={null} CONNECT TO jdbc:ocient://:/test; ``` Create a schema within the database to organize your tables. ```sql SQL theme={null} CREATE SCHEMA loading; ``` For details on managing databases and schemas, see [Databases](/databases) and [Schemas](/schemas). ## Step 3: Create a Table Create a table that covers all supported Ocient data types. This table serves as the target for the data pipeline in the next step. The `CREATE TABLE` SQL statement defines the column names, data types, and a on the `col_timestamp` column in the `data_type_coverage` table. The TimeKey partitions data by time so that the system can skip irrelevant segments during time-filtered queries. ```sql SQL theme={null} CREATE TABLE loading.data_type_coverage ( col_bigint BIGINT, col_binary BINARY(2), col_boolean BOOLEAN, col_char CHAR(64), col_date DATE, col_decimal DECIMAL(18, 4), col_double DOUBLE, col_float FLOAT, col_int INT, col_int_array INT[], col_ipv4 IPV4, col_ip IP, col_smallint SMALLINT, col_point POINT, col_linestring LINESTRING, col_polygon POLYGON, col_time TIME, col_timestamp TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, col_tinyint TINYINT, col_tuple TUPLE<> NULL, col_uuid UUID, col_varbinary VARBINARY, col_varchar VARCHAR ); ``` This table intentionally keeps the schema simple to focus on the loading workflow. In production, you should also define a Clustering Key and secondary indexes based on your expected query patterns. For details on table design options, see: * [TimeKeys and Clustering Keys](/timekeys-and-clustering-keys) — Segment keys that partition and order data for faster queries. * [Secondary Indexes](/secondary-indexes) — Additional indexes for columns used in filters. * [Table Compression Options](/table-compression-options) — Compression settings that reduce storage requirements. * [CREATE TABLE SQL Statement Examples](/create-table-sql-statement-examples) — Examples of table definitions for different use cases. ## Step 4: Create a Data Pipeline Data pipelines are the way to load data into an Ocient System. Each pipeline is a SQL object that defines the source, data format, and transformations for loading rows into one or more tables. For a full overview of pipeline concepts, see [Load Data](/load-data). ### Use a Sample Data Set The sample data is a set of gzip-compressed CSV files hosted in a public S3 bucket. Each file contains one header row followed by data rows with columns covering every Ocient data type. ### Create the Pipeline This data pipeline reads CSV files from the S3 source, parses each field using named headers, applies type conversions where needed, and loads the results into the `data_type_coverage` table. ```sql SQL theme={null} CREATE BATCH PIPELINE data_type_coverage_pipeline SOURCE S3 ENDPOINT 'https://s3.us-east-1.amazonaws.com' BUCKET 'ocient-docs' PREFIX 'all_data_types/small/' FILTER_GLOB '**output*.csv.gz' COMPRESSION_METHOD 'gzip' SORT_BY 'filename' EXTRACT FORMAT delimited RECORD_DELIMITER e'\n' FIELD_DELIMITER ',' FIELD_OPTIONALLY_ENCLOSED_BY '"' NUM_HEADER_LINES 1 HEADERS [ 'bigint', 'binary', 'boolean', 'char', 'date', 'decimal', 'double', 'float', 'int', 'int_array', 'ipv4', 'ip', 'smallint', 'point', 'linestring', 'polygon', 'time', 'timestamp', 'tinyint', 'tuple', 'uuid', 'varbinary', 'varchar' ] OPEN_ARRAY '[' CLOSE_ARRAY ']' ARRAY_ELEMENT_DELIMITER ',' OPEN_OBJECT '(' CLOSE_OBJECT ')' INTO test.loading.data_type_coverage SELECT $bigint AS col_bigint, $binary AS col_binary, $boolean AS col_boolean, $char AS col_char, $date AS col_date, $decimal AS col_decimal, DOUBLE(IF(LOWER($double) = 'nan',NULL, $double)) AS col_double, FLOAT(IF(LOWER($float) = 'nan', NULL, $float)) AS col_float, $int AS col_int, INT[]($int_array) AS col_int_array, $ipv4 AS col_ipv4, $ip AS col_ip, $smallint AS col_smallint, $point AS col_point, $linestring AS col_linestring, $polygon AS col_polygon, $time AS col_time, REPLACE(REPLACE($timestamp, 'T', ' '), 'Z', '') AS col_timestamp, $tinyint AS col_tinyint, $tuple AS col_tuple, $uuid AS col_uuid, VARBINARY(IF($varbinary IS NULL, NULL, IF(LENGTH($varbinary) % 2 = 1, CONCAT('0', $varbinary), $varbinary))) AS col_varbinary, $varchar AS col_varchar; ``` The pipeline has three sections: * `SOURCE` — Identifies the S3 bucket, file path prefix, and glob filter. The `COMPRESSION_METHOD` parameter indicates that the source files are gzip-compressed. * `EXTRACT` — Defines the CSV format options, including delimiters, quoting, and the header names that the `SELECT` clause references. The `OPEN_ARRAY` and `CLOSE_ARRAY`, and `OPEN_OBJECT` and `CLOSE_OBJECT` parameters tell the parser how array and tuple values are encoded in the CSV data. * `SELECT` — Maps each source field to a target column. Some fields require explicit type conversion. For example, `DOUBLE($double)` converts the extracted string to a `DOUBLE` value, and the nested `REPLACE` functions strip the `T` and `Z` characters from timestamps to match the Ocient `TIMESTAMP` format. This `VARBINARY` expression pads odd-length hex strings with a leading zero to ensure all rows that are not `NULL` convert to the `VARBINARY` data type. For details on pipeline syntax and options, see [Data Pipelines](/data-pipelines). For details on supported data formats, see [Data Formats for Data Pipelines](/data-formats-for-data-pipelines). For details on data type handling during loading, see [Data Types for Data Pipelines](/data-types-for-data-pipelines). For the functions, see [Transform Data in Data Pipelines](/transform-data-in-data-pipelines). ## Step 5: Start and Monitor the Pipeline Start the data pipeline to begin loading data using the `START PIPELINE` SQL statement. ```sql SQL theme={null} START PIPELINE data_type_coverage_pipeline; ``` Check the pipeline status while it runs. ```sql SQL theme={null} SHOW PIPELINE_STATUS; ``` Output ```none Text theme={null} database_name pipeline_name table_names status percent_complete records_processed records_loaded records_failed -------------- ------------------------------ --------------------------------- -------- ---------------- ------------------ --------------- --------------- test data_type_coverage_pipeline ["loading.data_type_coverage"] RUNNING 0.0 0 0 0 ``` Wait for the `status` column to show the `COMPLETED` status. You can execute the `SHOW PIPELINE_STATUS` statement again to check progress. For larger data sets, the `percent_complete`, `records_processed`, and `records_loaded` columns update as the pipeline progresses. For more detailed monitoring options, including system catalog tables and metrics endpoints, see [Monitor Data Pipelines](/monitor-data-pipelines). If the pipeline enters the `FAILED` status, see [Manage Errors in Data Pipelines](/manage-errors-in-data-pipelines) and [Data Pipeline Loading Errors](/data-pipeline-loading-errors). ## Step 6: Validate the Loaded Data After the pipeline completes, verify that the data loaded correctly before you begin analysis. ### Verify the Row Count Confirm that the number of loaded rows matches the pipeline metrics. ```sql SQL theme={null} SELECT COUNT(*) AS total_rows FROM loading.data_type_coverage; ``` Cross-reference this count with the `records_loaded` value from the `SHOW PIPELINE_STATUS` statement. ### Check the Time Range Verify that the `col_timestamp` column covers the expected date range. Gaps in the range can indicate missing source files. ```sql SQL theme={null} SELECT MIN(col_timestamp) AS earliest, MAX(col_timestamp) AS latest, COUNT(DISTINCT CAST(col_timestamp AS DATE)) AS distinct_days FROM loading.data_type_coverage; ``` ### Inspect for NULL Values Check for unexpected NULL values in columns that should be populated. ```sql SQL theme={null} SELECT COUNT(*) AS total_rows, COUNT(col_bigint) AS non_null_bigint, COUNT(col_double) AS non_null_double, COUNT(col_timestamp) AS non_null_timestamp, COUNT(col_varchar) AS non_null_varchar, COUNT(col_int_array) AS non_null_int_array, COUNT(col_tuple) AS non_null_tuple FROM loading.data_type_coverage; ``` If a column has significantly fewer non-NULL values than expected, review the source data and the pipeline `SELECT` transformations for parsing issues. ### Sample Records Review a small set of records to confirm that type conversions applied correctly using the `LIMIT` keyword. ```sql SQL theme={null} SELECT * FROM loading.data_type_coverage LIMIT 5; ``` ## Step 7: Query the Data With validated data in the table, you can begin running analytic queries. These examples demonstrate common query patterns across different data types. ### Aggregate Numeric Data Compute summary statistics on the numeric columns. In this case, retrieve the count of rows, the minimum number of the `col_int` column, the maximum number of the `col_int` column, and the average of the numbers in the `col_smallint` column. ```sql SQL theme={null} SELECT COUNT(*) AS total_rows, MIN(col_int) AS min_int, MAX(col_int) AS max_int, AVG(col_smallint) AS avg_smallint FROM loading.data_type_coverage; ``` ### Filter by Time Range Use the `col_timestamp` TimeKey column to filter data efficiently. The Ocient System skips segments outside the specified time range of January 2024. ```sql SQL theme={null} SELECT CAST(col_timestamp AS DATE) AS day, COUNT(*) AS row_count, COUNT(DISTINCT col_int) AS distinct_ints FROM loading.data_type_coverage WHERE col_timestamp >= TIMESTAMP '2024-01-01 00:00:00' AND col_timestamp < TIMESTAMP '2024-02-01 00:00:00' GROUP BY CAST(col_timestamp AS DATE) ORDER BY day; ``` ### Query Geospatial Data Inspect the geospatial columns loaded into the table for the first 10 rows using the `LIMIT` keyword. ```sql SQL theme={null} SELECT col_point, col_linestring, col_polygon FROM loading.data_type_coverage WHERE col_point IS NOT NULL LIMIT 10; ``` For details on geospatial functions and spatial queries, see [Geospatial Functions](/geospatial-functions). ### Explore Data Type Distributions Summarize the distinct values and NULL rates for each column type. ```sql SQL theme={null} SELECT COUNT(DISTINCT col_boolean) AS distinct_boolean, COUNT(DISTINCT col_tinyint) AS distinct_tinyint, COUNT(DISTINCT col_smallint) AS distinct_smallint, COUNT(DISTINCT col_int) AS distinct_int, COUNT(DISTINCT col_uuid) AS distinct_uuid FROM loading.data_type_coverage; ``` For a full reference of SQL functions, see [Functions Overview](/functions-overview). For tips on writing queries that leverage segment keys and indexes, see [Query Performance Tuning](/query-performance-tuning). ### Remove the Data Pipeline and Table After you finish exploring the sample data, remove the `data_type_coverage_pipeline` pipeline. ```sql SQL theme={null} DROP PIPELINE data_type_coverage_pipeline; ``` Remove the `data_type_coverage` table. ```sql SQL theme={null} DROP TABLE loading.data_type_coverage; ``` Removing a pipeline removes the pipeline metadata from the system but does not affect the data already loaded into the target table. Whereas removing the table removes the data. ## Related Links [Data Pipelines](/data-pipelines) [Tables](/tables) [Understanding Data Types](/understanding-data-types) [SQL Reference](/sql-reference) # Load Data Source: https://docs.ocient.com/load-data Overview of Ocient data pipelines for SQL-based ETL: real-time transformations, deduplication, parallel processing, and support for S3, Kafka, and HDFS sources. Loading data into the using the data pipeline functionality is as simple as writing a SQL statement to query data from different supported sources. The loading operation provides real-time transformations and maximizes performance using all available processing resources. With data pipelines, you manage data loading activities using Data Definition Language (DDL) statements similar to the ones used for managing database tables. You can access all information about pipelines by querying the system catalog tables. You can execute operations on pipelines with a command-line statement using standard interfaces like JDBC and ODBC. ## Data Pipelines Overview A data pipeline is the primary way that the System loads data. Each data pipeline is a database object that defines the end-to-end processing of data for the extraction, transformation, and load into Ocient tables. Data pipelines execute across your Ocient System to coordinate parallel loading tasks across many Loader Nodes and pipelines. ## The Pipeline Object Creating a pipeline is complex as there are many options to consider. Use the `PREVIEW PIPELINE` SQL statement to preview the pipeline creation and load of data to ensure that the pipeline definition returns the results you expect. After you are satisfied with the results, you can execute the `CREATE PIPELINE` statement to create the pipeline at scale. For details, see the [CREATE PIPELINE](/data-pipelines#create-pipeline) SQL statement. When you create a pipeline, you assign it a name and the pipeline exists as an object in the Ocient System connected to this name. Then, you can control the object using your chosen name with SQL statements like `START PIPELINE`, `STOP PIPELINE`, `CREATE OR REPLACE PIPELINE`, and `DROP PIPELINE`. You can also define your own function by using the `CREATE PIPELINE FUNCTION` SQL statement. Execute these statements using a SQL connection. A pipeline maintains its own position during a load and enforces deduplication logic to ensure that the Ocient System only loads data once. The lifecycle of a pipeline defines both the deduplication logic and load position. * Pipeline Events: During the life of a pipeline, you can start, stop, modify, and resume the pipeline without duplicating source data or losing the position in a load. * Pipeline Updates: You can modify pipelines using the `CREATE OR REPLACE` SQL statement to update the transforms in a pipeline, but maintain the current position in the load. * Load Position: If target tables are truncated or if a target table is dropped and then recreated, the pipeline maintains its own position in the load and continues from its last position. Pipelines also gracefully handles many error cases and system failure modes. The pipeline stores any errors that occur in the system catalog in association with the pipeline. For more details on Deduplication and Error Tolerance settings, see: * [Deduplication in Data Pipelines](/deduplication-in-data-pipelines) * [Error Tolerance in Data Pipelines](/error-tolerance-in-data-pipelines) * [START PIPELINE Reference](/data-pipelines#start-pipeline) ## Parts of a Pipeline Pipelines can operate in either a `BATCH` or `CONTINUOUS` mode based on how you plan to load your data. Or, for transactional batch loads, you can use the `TRANSACTIONAL` mode with the optional `START FOREGROUND` clause. The DDL for Data Pipelines has three sections. | **Section** | **Syntax** | | --------------- | ----------------------- | | Data Source | `SOURCE ...` | | Data Format | `EXTRACT FORMAT ...` | | Transformations | `INTO table SELECT ...` | Thus, for example, the full SQL syntax has this format for the `orders_pipeline` data pipeline. This syntax has an S3 data source and loads data into the `orders` table in the `public` schema. The `SELECT` SQL statement in the `CREATE PIPELINE` statement selects the `id`, `user_id`, `product_id`, and other columns from a table. ```sql SQL theme={null} CREATE PIPELINE orders_pipeline SOURCE S3 ENDPOINT '...' BUCKET '...' FILTER '...' EXTRACT FORMAT delimited ... INTO "public"."orders" SELECT $1 as id, $2 as user_id, $3 as product_id, ... ``` ### Data Source The `SOURCE` part in a pipeline defines the source of the data to load. This DDL part identifies the type of the source (e.g., S3, ) as well as the details for the data that is relevant to the source (e.g., S3 bucket and filters, Kafka topic and consumer configuration). * [Kafka Source Options](/data-pipelines#kafka-source-options) * [S3 Source Options](/data-pipelines#s3-source-options) * [File-based Source Options](/data-pipelines#file-based-source-options) * [SOURCE Options Reference](/data-pipelines#source-options) ### Data Format The `EXTRACT` part in a pipeline defines the format of the data that the Ocient System extracts from the data source. This part includes the data format (e.g., delimited, binary, JSON) and the details about the records (e.g., record delimiter, how to treat empty data). In addition, the Ocient System extracts metadata that you can load into tables. * [Loading Delimited and CSV Data](/data-formats-for-data-pipelines#load-delimited-and-csv-data) * [Loading JSON Data](/data-formats-for-data-pipelines#load-json-data) * [Load Parquet Data](/data-formats-for-data-pipelines#load-parquet-data) * [Loading Binary Data](/data-formats-for-data-pipelines) * [Loading Metadata Values](/data-formats-for-data-pipelines) * [EXTRACT Options Reference](/data-pipelines#extract-options) ### Transformations The `INTO table SELECT ...` SQL statement defines the target tables where the data loads and uses a `SELECT` SQL statement to extract fields from the source data (e.g., CSV field index `$5`, JSON selector `$order.user_name`) and map them to target columns (e.g., `... as my_column`). The `SELECT` statement in a pipeline can utilize a subset of SQL functions to convert the extracted data into the required format during loading (e.g., `TO_TIMESTAMP($my_date, 'YYYY-MM-DD')`). The selectors you use to extract data from the source records can differ based on the data format (e.g., `JSON` or `DELIMITED`). * [Supported Transformation Functions](/transform-data-in-data-pipelines) * [Use Source Field References](/transform-data-in-data-pipelines#source-field-references) * [Supported JSON Selectors](/data-formats-for-data-pipelines#supported-json-selectors) For loading workflows, see these examples: * [Data Pipeline Load of CSV Data from S3](/data-pipeline-load-of-csv-data-from-s3) * [Data Pipeline Load of JSON Data from Kafka](/data-pipeline-load-of-json-data-from-kafka) * [Data Pipeline Load of Parquet Data from S3](/data-pipeline-load-of-parquet-data-from-s3) ## Pipeline Lifecycle This state machine shows the state transitions for a data pipeline. You can start and stop data pipelines using DDL statements. The status of the pipeline transitions automatically as the pipeline completes all assigned work or reaches a failed state. While executing, the pipeline lists files or connects to streaming sources and creates tasks to execute work. These states reflect the overall progress of a pipeline and capture details of the underlying tasks that execute the work of the pipeline. | **State** | **Description** | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Created | The pipeline is created, but no user has started it.
\* Zero tasks in the `CREATED` state.
\* Zero files listed or no Kafka Consumer group created. | | Running | The pipeline is running after a user starts the pipeline.
\* At least one task is in the `QUEUED`, `RUNNING`, or `CANCELLING` states.
\* At least one file listed or Kafka Consumer group created. | | Stopped | The pipeline is not processing data after a user stops the pipeline.
\* All tasks are in the `QUEUED`, `COMPLETE`, `FAILED`, or `CANCELLED` states. | | Completed | The pipeline finished all assigned work according to the error limits defined for the pipeline.
\* All tasks in are in the `COMPLETE` state. | | Failed | The pipeline failed to complete and no longer runs due to error limits defined for the pipeline.
\* At least one task in the `FAILED` state.
\* All tasks are in the `QUEUED`, `COMPLETE`, `FAILED`, or `CANCELLED` states. | In this state diagram, the transition arrows display the user actions that can trigger a state change. State transitions without a label are system-initiated transitions. Pipeline status state machine with created, running, stopped, completed, and failed states ## Observe Pipelines During pipeline operation, all of the information you need to observe progress, pipeline status, key events, success or failure, and errors is available in system catalog tables in the Ocient System. In addition, performance counters are available for many key loading metrics that you can add to observability systems. * [Monitor Data Pipelines](/monitor-data-pipelines) * [System Catalog](/system-catalog) ## Troubleshooting Pipelines Data pipelines support robust error-handling capabilities crucial for developing new pipelines, identifying issues with ongoing operations, and correcting bad data. Pipelines can include a bad data target where the Ocient System saves bad data for troubleshooting. In addition, the system captures errors that you encounter during loading in system catalog tables such as `sys.pipeline_events` and `sys.pipeline_errors`. Exploring the data in these tables enables you to detect issues and identify the root cause of errors during loading. * [Manage Errors in Data Pipelines](/manage-errors-in-data-pipelines) * [Frequently Asked Questions for Data Pipelines](/frequently-asked-questions-for-data-pipelines) * [Data Pipeline Loading Errors](/data-pipeline-loading-errors) * [Bad Data Targets Reference](/data-pipelines) * [Error Handling Options Reference](/data-pipelines) ## Loading Architecture Pipelines require Loader Nodes to operate. Loader Nodes are Ocient Nodes that have the `streamloader` role assignment. The loading system can use all active nodes with this role to process pipelines. When you execute a pipeline, Ocient spreads the work among the available Loader Nodes. The system executes these parallel processes as tasks on each of the Loader Nodes. Then, the system partitions the files or Kafka partitions to load across the tasks, and loading proceeds in parallel. When all tasks are completed, the pipeline is completed. Many pipelines can run in parallel on the Loader Nodes, and these Nodes share resources. The scale-out architecture of the Ocient System allows you to add more Loader Nodes as needed to expand the volume and complexity of pipelines that you can operate at the throughput required by your application. Ocient loading architecture that consists of SQL client connections that interface with SQL Nodes, data sources that interface with Loader Nodes, and SQL and Loader Nodes interface with Foundation Nodes for storage and processing. ## Internal Operations of a Pipeline As pipelines process records from a data source, the Ocient System builds these rows into a column-oriented storage structure called a page. The system optimizes pages for rapid ingestion of data. The system stores pages on Foundation Nodes to ensure data integrity. As pages accumulate, Loader Nodes convert pages into a columnar storage format named a Segment. Segments are highly compressed data structures that include data, indexes, and other statistical metadata to optimize query performance. After the Ocient System creates Segments, the system forms them into groups and transfers them to Foundation Nodes for storage. You do not typically need to be concerned with or control these internal operations. However, it is important to understand that pipeline throughput can be affected by the parallel processing of many pipelines on shared resources, the number of indexes, and the type of compression used on your tables. ## Related Links [Pipeline Privileges Reference](/data-control-language-dcl-statement-reference#data-pipeline-privileges) [Data Pipelines Reference](/data-pipelines) [PREVIEW PIPELINE](/data-pipelines#preview-pipeline) [CREATE PIPELINE](/data-pipelines) [DROP PIPELINE](/data-pipelines#drop-pipeline) [START PIPELINE](/data-pipelines) [STOP PIPELINE](/data-pipelines#stop-pipeline) [ALTER PIPELINE RENAME](/data-pipelines#alter-pipeline-rename) [EXPORT PIPELINE](/data-pipelines#export-pipeline) [CREATE PIPELINE FUNCTION](/data-pipelines#create-pipeline-function) [ Transactional Data Pipelines ](/transactional-data-pipelines) [Frequently Asked Questions for Data Pipelines](/frequently-asked-questions-for-data-pipelines) # Load Data from External Sources in Data Pipelines Source: https://docs.ocient.com/load-data-from-external-sources-in-data-pipelines Use the LOOKUP keyword and LOOKUP function in CREATE PIPELINE statements to enrich data loads in Ocient by joining records with external source tables. The data pipeline functionality in the System enables you to load data from external sources, such as other databases, using the `LOOKUP` function and the `LOOKUP` keyword of the corresponding `CREATE PIPELINE` SQL statement. You must use the function and the `LOOKUP` keyword together to join data from an external source. You can use the `LOOKUP` function with multiple external sources by specifying each source with its `LOOKUP` function and corresponding `LOOKUP` keyword syntax. **Syntax** The `LOOKUP` function returns a value from an external source table based on a join between a specified value and another column in the source table. The function creates and executes this SQL statement from the specified function arguments. ```sql SQL theme={null} SELECT return_column_name FROM lookup_source_name WHERE value = join_column_name; ``` The source table should only have unique values for the `join_column_name` column for the join operation. ```sql SQL theme={null} LOOKUP(lookup_source_name, value, join_column_name, return_column_name) ``` | **Argument** | **Data** **Type** | **Description** | | -------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `lookup_source_name` | VARCHAR | The name of the external source defined by the `LOOKUP` keyword in the `CREATE PIPELINE` SQL statement. | | `value` | VARCHAR, BOOLEAN, BYTE, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, DATE, TIME, TIMESTAMP, HASH, BINARY, DECIMAL, UUID | The value for the lookup. | | `join_column_name` | VARCHAR | The name of a column in the external source table for the join operation. | | `return_column_name` | VARCHAR, BOOLEAN, BYTE, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, DATE, TIME, TIMESTAMP, HASH, BINARY, DECIMAL, UUID | The name of the column in the external source table that contains the data to load. | If the data type of the returned value of the `LOOKUP` function is not one of the data types specified for the `return_column_name` argument, then the function transforms the value to the nearest compatible type, typically `VARCHAR`. **Example** Create a data pipeline `lookup_data_pipeline` that loads data from the CSV file `data.csv`. Use an external source `existing_all_types` with the table `tablename` in the schema `schemaname` using a JDBC connection with the connection string: `jdbc:host://111.1.1.1:4200/databasename;user=username@databasename;password=testpassword`. For your connection string, substitute these variables with the values specific to your database and credentials: * `host` — Hostname * `111.1.1.1` — IP address * `4200` — Port number * `databasename` — Database name * `username` — Username * `testpassword` — Password Look up data in the `col_binary` column of the `tablename` table by joining the second column of the CSV file to the `col_bigint` column in the `tablename` table. Load the data in the `col_binary` column based on the result of the `LOOKUP` function. ```sql SQL theme={null} CREATE PIPELINE lookup_data_pipeline SOURCE FILESYSTEM FILTER '/tmp/folder/data.csv' LOOKUP existing_all_types CONNECTION_TYPE 'jdbc' CONNECTION_STRING 'jdbc:host://111.1.1.1:4200/databasename;user=username@databasename;password=testpassword' LOOKUP_SCHEMA 'schemaname' LOOKUP_TABLE 'tablename' EXTRACT FORMAT CSV INTO public.destination_table SELECT $1 AS col_pk, LOOKUP('existing_all_types', $2, 'col_bigint', 'col_binary') AS col_binary; ``` ## Performance Considerations The performance of the lookup operation depends on the size of the external source. If the external source is large, the lookup operation is slower. The Ocient System recommends keeping the size to less than 5 million rows for the total size of all external source tables used by actively running data pipelines in the system. ## Related Links [Load Data](/load-data) [CREATE PIPELINE](/data-pipelines#create-pipeline) [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) # Load Data into Multiple Targets in Data Pipelines Source: https://docs.ocient.com/load-data-into-multiple-targets-in-data-pipelines Use one Ocient data pipeline to load time-partitioned data from AWS S3 into multiple target tables in batches for parallel ingestion and routing. A common setup for loading files in a batch into is to load from a bucket on S3 with time-partitioned data. Often, you must perform a batch load repeatedly to load new files. Ocient uses data pipelines to transform each document into rows in one or more different tables. The loading and transformation capabilities use a simple SQL-like syntax for transforming data. This tutorial guides you through a load into multiple target tables using a small data set in CSV format. The data in this example comes from a test set for the Business Intelligence tool. ## Prerequisites This tutorial assumes that: 1. The Ocient System has network access to S3 from the Loader Nodes. 2. An Ocient System is installed and configured with an active Storage Cluster (see the [Ocient Application Configuration](/ocient-application-configuration) guide). ## Step 1: Create a New Database Connect to a SQL Node using the [Commands Supported by the Ocient JDBC CLI Program](/commands-supported-by-the-ocient-jdbc-cli-program). Then, execute the `CREATE DATABASE` SQL statement for a database named `metabase`. ```sql SQL theme={null} CREATE DATABASE metabase; ``` ## Step 2: Create Tables Create two target tables for loading data for two different products. Create the `product129` table in the new database. First, connect to that database (e.g., `connect to jdbc:ocient://sql-node:4050/metabase`), and then execute this `CREATE TABLE` SQL statement that specifies to create a table with these columns and a clustering index based on the `user_id` and `product_id` columns: * `created_at` as a timestamp that is not nullable. * `id`, `user_id`, and `product_id` as integers that are not nullable. * `subtotal`, `tax`, `total`, and `discount` as floating point numbers. * `quantity` as an integer. ```sql SQL theme={null} CREATE TABLE public.product129 ( created_at TIMESTAMP TIME KEY BUCKET(30, DAY) NOT NULL, id INT NOT NULL, user_id INT NOT NULL, product_id INT NOT NULL, subtotal DOUBLE, tax DOUBLE, total DOUBLE, discount DOUBLE, quantity INT, CLUSTERING INDEX idx01 (user_id, product_id) ); ``` Create the second `product161` table in the same way. ```sql SQL theme={null} CREATE TABLE public.product161 ( created_at TIMESTAMP TIME KEY BUCKET(30, DAY) NOT NULL, id INT NOT NULL, user_id INT NOT NULL, product_id INT NOT NULL, subtotal DOUBLE, tax DOUBLE, total DOUBLE, discount DOUBLE, quantity INT, CLUSTERING INDEX idx01 (user_id, product_id) ); ``` The database creates the `product129`and `product161` tables, and you can begin loading data. ## Step 3: Create a Data Pipeline Create data pipelines using the `CREATE PIPELINE` SQL statement. To load data, you first create a pipeline with the definition of the source, data format, and transformation rules using a SQL-like declarative syntax. Then, you execute the `START PIPELINE` SQL statement to start the load. You can observe progress and status using system tables and views. Each Ocient pipeline defines a single data source and the target table or tables into which data loads. A data source includes the location of the source and filters on the source to define the specific data set to load. This example loads data from a data source located in a directory within the S3 bucket. First, inspect the data that you plan to load. Each document has a format similar to this example CSV file named `orders.csv`. ```none Text theme={null} id,user_id,product_id,subtotal,tax,total,discount,created_at,quantity 1,1,14,37.65,2.07,39.72,null,2019-02-11T21:40:27.892Z,2 2,1,123,110.93,6.1,117.03,null,2018-05-15T08:04:04.580Z,3 3,1,105,52.72,2.9,49.2,6.42,2019-12-06T22:22:48.544Z,2 ... ``` In this case, Ocient automatically transforms the data to the target columns using some sensible conventions. In other cases, loads require some transformation. Most transformations are identical to functions that already exist in the SQL dialect of the Ocient System. Create a pipeline named `orders_pipeline` for the orders data set from your database connection prompt. Use the S3 data source with endpoint `https://s3.us-east-1.amazonaws.com`, bucket `ocient-docs`, and filter `metabase_samples/csv/orders.csv`. Specify the CSV format with one header line. Load the data into the two target tables. The `SELECT` part of the SQL statement maps the fields in the CSV file to the target columns in the created table. ```sql SQL theme={null} CREATE PIPELINE orders_pipeline SOURCE S3 ENDPOINT 'https://s3.us-east-1.amazonaws.com' BUCKET 'ocient-docs' FILTER 'metabase_samples/csv/orders.csv' EXTRACT FORMAT csv NUM_HEADER_LINES 1 INTO public.product129 SELECT $1 as id, $2 as user_id, $3 as product_id, $4 as subtotal, $5 as tax, $6 as total, $7 as discount, $8 as created_at, $9 as quantity WHERE $3 = 129 INTO public.product161 SELECT $1 as id, $2 as user_id, $3 as product_id, $4 as subtotal, $5 as tax, $6 as total, $7 as discount, $8 as created_at, $9 as quantity WHERE $3 = 161; ``` The pipeline has three main sections: * `SOURCE` — In this case, load data from S3. Specify the endpoint and bucket. The `FILTER` parameter identifies the file or files to load. The load uses a single CSV file. Options exist to add wildcards or regular expressions to isolate different file sets. * `EXTRACT` — Set the format to CSV files and note that there is one header line in the file. This specification skips that row when the Ocient System processes the file. Many other options exist for delimited data such as a record delimiter and field delimiter. * `INTO ... SELECT` — Choose the target tables `public.product129` and `public.product161` and select the fields from the CSV file. The numeric index identifies each file field. Importantly, similar to other SQL syntax, the first field in the file is `$1`, not `$0`. Each field maps to a target column using the `as` syntax. The `WHERE` filter instructs the data pipeline to load order data for only two products, one product for each table. After you successfully create the `orders_pipeline` pipeline, execute the `START PIPELINE` SQL statement. ```sql SQL theme={null} START PIPELINE orders_pipeline; ``` ## Step 4: Observe the Load Progress With your pipeline running, data immediately begins to load from the S3 files that you defined. If there are many files in each file group, the load process first sorts the files into batches, partitions them for parallel processing, and assigns them to Loader Nodes. You can check the pipeline status and progress by querying `information_schema.pipeline_status` or by executing `SHOW PIPELINE_STATUS`. ```sql SQL theme={null} SHOW PIPELINE_STATUS; ``` *Output* ```sql SQL theme={null} database_name pipeline_name table_names status status_message percent_complete duration_seconds files_processed files_failed files_remaining records_processed records_loaded records_failed -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- metabase orders_pipeline ["public.product129" "public.product161"] RUNNING Started processing pipeline orders_pipeline 0.0 2.025824 0 0 1 0 0 0 ``` After the status of the pipeline changes to `COMPLETED`, all data is available in the target table. After a few seconds, the data is available for query in the `public.product129` table. ```sql SQL theme={null} SELECT COUNT(*) FROM public.product129; ``` Output ```sql SQL theme={null} count(*) -------------------- 93 ``` The data is also available for query in the `public.product161` table. ```sql SQL theme={null} SELECT COUNT(*) FROM public.product161; ``` Output ```sql SQL theme={null} count(*) -------------------- 92 ``` You can drop the pipeline with the `DROP PIPELINE orders_pipeline;` SQL statement. Execution of this statement leaves the data in your target table, but removes metadata about the pipeline execution from the system. ## Related Links [Data Pipelines Reference](/data-pipelines) [Load Delimited and CSV Data](/data-formats-for-data-pipelines#load-delimited-and-csv-data) [Data Types for Data Pipelines](/data-types-for-data-pipelines) [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) [Manage Errors in Data Pipelines](/manage-errors-in-data-pipelines) # Load Data with a User-Defined Data Pipeline Function Source: https://docs.ocient.com/load-data-with-a-user-defined-data-pipeline-function Load data into Ocient with user-defined data pipeline functions written in Java to handle custom source formats, transformations, and business logic. The data pipeline functionality in the System enables you to transform data during loading. You can create your own data pipeline function to perform a custom transformation or calculation. This example uses a user-defined data pipeline function to load weather data for Chicago. The function converts temperatures from Fahrenheit to Celsius. The formula for the conversion is `°C = (°F - 32) × 5/9`. The precipitation type data is an array with zero or more values in lowercase. The data pipeline also utilizes a lambda function to convert the first character of each precipitation type to uppercase. The pipeline uses the metadata `filename` in the data load as an audit trail.  ## Retrieve the Weather Data from the Source Retrieve year-to-date weather data for Chicago in JSON format from the [Weather Query Builder](https://www.visualcrossing.com/weather-query-builder/). Here is a sample row of data. The data contains: * `queryCost` — Cost of the query * `latitude` — Latitude of the location * `longitude` — Longitude of the location * `resolvedAddress` — Address of the location with the country * `address` — City and state of the address * `timezone` — Time zone of the location * `tzoffset` — Time zone offset * `name` — Name of the location * `days` — Daily weather data in an array with these fields: * `datetime` — Date of the weather data * `datetimeEpoch` — Date epoch of the weather data * `tempmax` — Maximum temperature of the day * `tempmin` — Minimum temperature of the day * `temp` — Temperature of the day * `feelslike` — Feels like temperature of the day * `humidity` — Humidity for the day * `precip` — Precipitation amount for the day * `precipprob` — Probability of precipitation for the day * `preciptype` — Type of precipitation for the day (array of strings for types such as rain, snow, and so on) * `windspeed` — Wind speed for the day ```json JSON theme={null} {    "queryCost" : 212,    "latitude" : 41.8843,    "longitude" : -87.6324,    "resolvedAddress" : "Chicago, IL, United States",    "address" : "chicago, il",    "timezone" : "America/Chicago",    "tzoffset" : -6.0,    "name" : "chicago, il",    "days" : [ {      "datetime" : "2025-01-01",      "datetimeEpoch" : 1735711200,      "tempmax" : 35.3,      "tempmin" : 24.8,      "temp" : 30.0,      "feelslike" : 19.5,      "humidity" : 63.9,      "precip" : 0.037,     "precipprob" : 100.0,      "preciptype" : [ "rain", "snow" ],      "windspeed" : 17.6  } ] } ``` Create the `chicago_weather_ytd` table with these columns: * `latitude` — Latitude of the location * `longitude` — Longitude of the location * `location_name` — Name of the location * `log_date` — Date of the weather event * `temp` — Temperature of the day * `preciptype` — An array of characters for multiple types of weather events * `file_name` — Name of the file that contains the weather data ```sql SQL theme={null} CREATE TABLE chicago_weather_ytd( "latitude" double,    "longitude" double,    "location_name" varchar(20),    "log_date" date,    "temp" double,    "preciptype" varchar(100)[],            "file_name" varchar(500));  ``` ## Create a User-Defined Function for Temperature Conversion Create a user-defined function that converts a degree value from Fahrenheit to Celsius, and round the returned value to one decimal place. Use code to specify the calculation and rounding. ```sql SQL theme={null} CREATE PIPELINE FUNCTION f_to_c(f_value DOUBLE NOT NULL)  LANGUAGE GROOVY  RETURNS DOUBLE NOT NULL  IMPORTS [] AS $$  return (10 * ((f_value - 32.0) * 5.0 / 9.0)).toInteger() / 10   $$;  ``` ## Create a Data Pipeline to Load the Weather Data Build a pipeline by using a preview of the data to load with the `PREVIEW PIPELINE` SQL statement to see how a limited number of rows loads. Preview four rows. Specify the JSON format. Transform the latitude and longitude to the double type using the `DOUBLE` cast function. For the date of the weather of the event `log_date`, expand the input array of days into its elements using the `EXPLODE_OUTER` function and convert the date to a `date` type using the `DATE` cast function. For the temperature of the day, expand the input array of days into its elements using the `EXPLODE_OUTER` function and convert the temperature to the double type using the `DOUBLE` cast function. Execute the `f_to_c` user-defined function to convert the temperature from Fahrenheit to Celsius. For the type of precipitation, expand the input array of days into its elements using the `EXPLODE_OUTER` function. Then, transform each element of the input array using a lambda function that converts the first character of the precipitation type to uppercase. Use the `METADATA` function to capture the filename. ```sql SQL theme={null} PREVIEW PIPELINE chicago_weather_ytd_pipeline  SOURCE filesystem  FILTER '/tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json'  LIMIT 4 EXTRACT FORMAT json   INTO chicago_weather_ytd  SELECT DOUBLE($latitude) AS latitude,    DOUBLE($longitude) AS longitude,    $address AS location_name,    DATE(EXPLODE_OUTER($days[].datetime)) AS log_date,  f_to_c(DOUBLE(EXPLODE_OUTER($days[].temp))) AS temp,  EXPLODE_OUTER(TRANSFORM($days[].preciptype[],  (x VARCHAR) -> CONCAT(UPPER(SUBSTRING(x,1,1)),SUBSTRING(x,2,20)))) AS preciptype,  METADATA('filename') AS file_name;  ``` Output ```none Text theme={null} latitude              longitude             location_name                                log_date   temp                  preciptype                                                                      file_name  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  41.8843               -87.6324              chicago, il                                  2025-01-01 -1.1    [Rain,Snow]                                                                    /tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json  41.8843               -87.6324              chicago, il                                  2025-01-02 -2.2    [Rain,Snow]                                                                    /tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json  41.8843               -87.6324              chicago, il                                  2025-01-03 -4.6     [Snow]                                                                         /tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json  41.8843               -87.6324              chicago, il                                  2025-01-04 -7.2     NULL                                                                           /tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json  ``` Create the data pipeline by replacing `PREVIEW` with the `CREATE` keyword in the SQL statement. ```sql SQL theme={null} CREATE PIPELINE chicago_weather_ytd_pipeline SOURCE filesystem  FILTER '/tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json'  EXTRACT FORMAT json   INTO chicago_weather_ytd  SELECT DOUBLE($latitude) AS latitude,    DOUBLE($longitude) AS longitude,    $address AS location_name,    DATE(EXPLODE_OUTER($days[].datetime)) AS log_date,  f_to_c(DOUBLE(EXPLODE_OUTER($days[].temp))) AS temp,  EXPLODE_OUTER(TRANSFORM($days[].preciptype[],  (x VARCHAR) -> CONCAT(UPPER(SUBSTRING(x,1,1)),SUBSTRING(x,2,20)))) AS preciptype,  METADATA('filename') AS file_name;  ``` ## Execute the Data Pipeline Load Start the data pipeline by allowing 100 errors before the pipeline fails. The data pipeline runs to completion. ```sql SQL theme={null} START PIPELINE chicago_weather_ytd_pipeline ERROR LIMIT 100;  ``` Use the `SHOW PIPELINE_STATUS` command to view the status of the data pipeline. ```sql SQL theme={null} SHOW PIPELINE_STATUS; ``` Output ```none Text theme={null} database_name|pipeline_name |table_names |status |status_message |duration_seconds|files_processed|files_failed|files_remaining|files_total|fraction_complete|records_processed|records_loaded|records_failed| -------------+-------------------+------------------------------------+---------+--------------------------------------------------+----------------+---------------+------------+---------------+-----------+-----------------+-----------------+--------------+--------------+ training |chicago_weather_ytd|{'admin@system.chicago_weather_ytd'}|COMPLETED|Completed processing pipeline chicago_weather_ytd.| 11| 1| 0| 0| 1| 1.0| 212| 212| 0| training |city_weather |{'admin@system.city_weather'} |COMPLETED|Completed processing pipeline city_weather. | 10| 1| 0| 0| 1| 1.0| 48| 48| 0| training |counties_load |{'admin@system.us_counties'} |COMPLETED|Completed processing pipeline counties_load. | 11| 1| 0| 0| 1| 1.0| 3429| 3429| 0| ``` Display five rows of the loaded data sorted by the date of the weather event. ```sql SQL theme={null}  SELECT *  FROM chicago_weather_ytd  LIMIT 5  ORDER BY log_date;  ``` ```none Text theme={null} latitude|longitude|location_name|log_date |temp|preciptype |file_name | --------+---------+-------------+----------+----+---------------+---------------------------------------------------------------+ 41.8843| -87.6324|chicago, il |2025-01-01|-1.1|{'Rain','Snow'}|/tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json| 41.8843| -87.6324|chicago, il |2025-01-02|-2.1|{'Rain','Snow'}|/tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json| 41.8843| -87.6324|chicago, il |2025-01-03|-4.6|{'Snow'} |/tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json| 41.8843| -87.6324|chicago, il |2025-01-04|-7.2|NULL |/tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json| 41.8843| -87.6324|chicago, il |2025-01-05|-6.4|{'Snow'} |/tmp/data/json_files/nested_json/Chicago_Weather_Daily_YTD.json| ``` ## Related Links [Data Pipelines](/data-pipelines) [User-Defined Data Pipeline Functions](/transform-data-in-data-pipelines#user-defined-data-pipeline-functions) [Lambda Functions](/transform-data-in-data-pipelines#lambda-functions) [Load Metadata and File-Based Partitioned Data in Data Pipelines](/load-metadata-and-file-based-partitioned-data-in-data-pipelines) # Load Geospatial Data in Data Pipelines Source: https://docs.ocient.com/load-geospatial-data-in-data-pipelines Load geospatial data into Ocient with data pipelines using OcientGeo functions such as ST_POINT, ST_LINESTRING, and ST_POLYGON for spatial analytics. Data pipeline loading supports the load of geospatial data for use with functionality in the . Use geospatial functions to manipulate this data. For details, see [Geospatial Functions](/geospatial-functions). ## Supported Geospatial Functions These geospatial functions are supported for data pipeline loading: * [ST\_FORCECCW](/polygon-constructors#st_forceccw) * [ST\_POINT](/point-constructors#st_point) or [ST\_MAKEPOINT](/point-constructors#st_makepoint) * [ST\_POINTFROMTEXT](/point-constructors#st_pointfromtext) * [ST\_POINTFROMEWKT](/point-constructors#st_pointfromewkt) * [ST\_LINESTRING](/linestring-constructors#st_linestring) * [ST\_LINEFROMTEXT](/linestring-constructors#st_linefromtext) * [ST\_MAKELINE](/linestring-constructors#st_makeline) * [ST\_LINEFROMEWKT](/linestring-constructors#st_linefromewkt) * [ST\_POLYGON(char)](/polygon-constructors#st_polygon-char) * [ST\_POLYGON(geo)](/polygon-constructors#st_polygon-geo) * [ST\_POLYGON](/polygon-constructors#st_polygon) or [ST\_MAKEPOLYGON](/polygon-constructors#st_makepolygon) * [ST\_POLYGONFROMTEXT](/polygon-constructors#st_polygonfromtext) * [ST\_POLYGONFROMEWKT](/polygon-constructors#st_polygonfromewkt) ## Data Pipeline Loading of Geospatial Data Considerations ### Auto-Casting Behavior of Source Data Types The System automatically casts geospatial data from the `CHAR` type to `POINT`, `LINESTRING`, or `POLYGON` types using the [ST\_POINTFROMTEXT](/point-constructors), [ST\_LINEFROMTEXT](/linestring-constructors), or [ST\_POLYGONFROMTEXT](/polygon-constructors) functions, respectively. The data must be in WKT format for the casting to work. If your data is not in the WKT format, use one of the supported geospatial functions to transform your data to the required type. ### Point Data Normalization During Data Pipeline Loading During loading, the Ocient System automatically performs normalization of point data into a regular format used within Ocient. The Ocient System performs these operations on point data during the load: * Constrain longitude to \[-180, 180] and latitude to \[-90, 90]. * Wrap around invalid coordinates using correct geographical handling. * Snap points near the pole to the pole. * Set the longitude of points on the pole to 0. * Remove signed zeros from coordinates, so -0 becomes 0. ### Polygon Rotation Convention The Ocient System has a standardized polygon orientation convention. The standard convention has the polygon with a counterclockwise rotation of the outer polygon ring and a clockwise rotation of the inner polygon ring. Use the [ST\_FORCECCW](/polygon-constructors) function to convert polygons to the standardized convention from systems that interpret polygon rotation differently. ### Size Limit for Geospatial Data Types The maximum allowed size of geospatial data types is 512 MiB. ## Examples of Loading Geospatial Data Use these examples to understand how to create a data pipeline to load geospatial data for different formats (point and WKT data). ### Load Geospatial Data Using Point Data Create the target table `cities` using the `CREATE TABLE` SQL statement with these columns: * `id` — Integer * `name` — Variable-length string * `location` — Numeric point ```sql SQL theme={null} CREATE TABLE cities ( id INT, name VARCHAR, location ST_POINT ); ``` Create the `cities` data pipeline using the `CREATE PIPELINE` SQL statement for an S3 source with bucket `ocient-docs`, filter for the folder `metabase_samples/jsonl/cities.jsonl`, and region `us-east-1`. Use the JSON format to load a JSON file into the `cities` table. Specify a numeric point using the [ST\_POINT](/point-constructors) function with the longitude and latitude coordinates. ```sql SQL theme={null} CREATE PIPELINE cities SOURCE S3 BUCKET 'ocient-docs' FILTER 'metabase_samples/jsonl/cities.jsonl' REGION 'us-east-1' EXTRACT FORMAT json INTO cities SELECT $id AS id, $name AS name, ST_POINT($longitude, $latitude) AS location; ``` ### Load Geospatial Data in WKT Format Create the target table `geospatial_table` using the `CREATE TABLE` SQL statement with these non-nullable columns: * `ts` — with daily time bucket as a timestamp * `row_number` — Variable-length string * `point` — POINT object * `linestring` — LINESTRING object * `polygon` — POLYGON object ```sql SQL theme={null} CREATE TABLE geospatial_table( ts TIMESTAMP TIME KEY BUCKET(1, DAY) NOT NULL, row_number INT NOT NULL, point ST_POINT NOT NULL, linestring ST_LINESTRING NOT NULL, polygon ST_POLYGON NOT NULL ); ``` Create the `geospatial_pipeline` data pipeline using the `CREATE PIPELINE` SQL statement for an S3 source with bucket `ocient-docs`, filter for the folder `gis_small/gis_types/*.csv`, and region `us-east-1`. Use the CSV format to load a CSV file into the `geospatial_table` table. The file contains geospatial data in WKT format in the sixth, ninth, and twelfth columns with POLYGON, LINESTRING, and POINT data, respectively. Specify the polygon data in WKT format by using the [ST\_POLYGONFROMTEXT](/polygon-constructors) function and then reorient the resulting polygon counter-clockwise for the exterior and clockwise for the interior by using the [ST\_FORCECCW](/polygon-constructors) function. During the load, the Ocient System automatically casts the WKT-formatted data in the three columns to the corresponding geospatial data types. ```sql SQL theme={null} CREATE PIPELINE geospatial_pipeline SOURCE s3 BUCKET 'ocient-docs' FILTER 'gis_small/gis_types/*.csv' REGION 'us-east-1' EXTRACT FORMAT csv INTO geospatial_table SELECT $1 AS ts, $2 AS row_number, $12 AS point, $9 AS linestring, ST_FORCECCW(ST_POLYGONFROMTEXT($6)) AS polygon; ``` ## Related Links [Geospatial Functions](/geospatial-functions) [Load Data](/load-data) # Load metadata and partitioned data in data pipelines Source: https://docs.ocient.com/load-metadata-and-file-based-partitioned-data-in-data-pipelines Use the METADATA() function in Ocient data pipelines to extract file metadata such as filenames and creation timestamps from source records during loading. The data pipeline loading infrastructure enables the loading of certain metadata values of a source record into the System. Metadata represents information about a record that is not represented in the record data itself, e.g., the filename of the source record or the timestamp when a record was created in . You can also load data in partitioned files using naming standards with a filter set in the file path. **Syntaxes** The `METADATA` function can load a variety of metadata along with each record of the data pipeline, or it can use key-value pairs to return the value in the filename metadata. **Load Metadata into a Column** To load a metadata value into a column, use the `METADATA` function in the `SELECT` SQL statement and replace `key` with the key you want to load. You can use transformation functions to achieve the final data type or allow the pipeline to apply automatic casting. ```sql SQL theme={null} METADATA(key) ``` | **Argument** | **Data** **Type** | **Description** | | ------------ | ----------------- | ---------------------------------------------------------- | | `key` | string | The name of the specified metadata key. Example: `'topic'` | **Retrieve a Value from Filename Metadata** To retrieve a value from a key-value pair in the filename metadata, specify the first argument as `'hive_partition'` and provide a search string for the key. In this case, the filename must follow Hive naming standards. The standard embeds field names and key-value pairs in path segments, such as `/year=2019/month=2/data.parquet`. If there are duplicate key-value pairs in the filename, the function uses the last pair. The function returns a string with the value of the associated key, and returns NULL if the file does not contain the search string or the filename does not follow Hive naming standards. ```sql SQL theme={null} METADATA('hive_partition',search_string) ``` | **Argument** | **Data** **Type** | **Description** | | --------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `search_string` | string | The search string to use as a key for searching the filename metadata to return the value from the key-value pair as a string. | Loads a variety of header metadata along with each record of the data pipeline. ```sql SQL theme={null} METADATA('header', header_row_index, header_col_index) ``` | **Argument** | **Data** **Type** | **Description** | | ------------------ | ----------------- | ----------------------------------------------------------- | | `header_row_index` | integer | The index, which starts at 1, of the header row to load. | | `header_col_index` | integer | The index, which starts at 1, of the header column to load. | Metadata fields cannot be combined in a transformation function with other source data. **Supported Values for the Metadata Key** The metadata values you can load differ based on the source type. This table lists the available values for the metadata keys specified using the `key` argument by source type and their returned data type. Metadata key values are not case-sensitive. | **Metadata Key** | **Source Type** | **Returned Value Data Type** | **Description** | | ------------------------- | ------------------------ | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `row_id` | File Load and Kafka Load | UUID | A unique row identifier that is generated by the Ocient System during loading. The Ocient System guarantees this identifier is unique for each loaded record because the generation is based on internal mechanisms that enforce record deduplication. | | `filename` | File Load | VARCHAR | The filename and full file path indicate the source of the record. | | `file_modified_timestamp` | File Load | TIMESTAMP | The timestamp that indicates when the file was last modified on the data source. | | `header` | Delimited File Load | VARCHAR | Returns the string representation of the header field for the specified header row and column.

Returns NULL if any of these conditions exist:

The specified field is not present.

The specified field is empty, and the `EMPTY_FIELD_AS_NULL` delimited extract option is set to `true`.

The specified field matches one of the strings in the `NULL_STRINGS` delimited extract option. | | `key` | Kafka Load | VARBINARY | The bytes of the Kafka key for the record. | | `line_number` | File Load | BIGINT | The line number in the file for the source of the record. | | `offset` | Kafka Load | BIGINT | The Kafka offset of the record. | | `partition` | Kafka Load | BIGINT | The name of the Kafka partition that indicates the source of the record. | | `record` | File and Kafka Load | VARCHAR | The string representation of the full record. For delimited, JSON, and binary formats, the system represents the record in the same way as the source. For XML, ASN.1, and formats, the system represents the record as a JSON object. | | `record_timestamp` | Kafka Load | TIMESTAMP | The timestamp that indicates when the record was created in the Kafka Broker. | | `source_record_id` | File and Kafka Load | UUID | A unique source row identifier that is generated by the Ocient System during loading. The Ocient System guarantees this unique identifier for each source record with a specific SOURCE and EXTRACT configuration because the generation is based on source names, source record numbers, and source record offsets. | | `topic` | Kafka Load | VARCHAR | The name of the Kafka topic that indicates the source of the record. | **Examples** **Load Filename Metadata** This example snippet uses the `CREATE PIPELINE` SQL statement to load data and the source filename with each record in a pipeline. For details about this pipeline definition and setup, see [Data Pipeline Load of CSV Data from S3](/data-pipeline-load-of-csv-data-from-s3). ```sql SQL theme={null} CREATE PIPELINE orders_pipeline SOURCE S3 ENDPOINT 'https://s3.us-east-1.amazonaws.com' BUCKET 'ocient-docs' FILTER 'metabase_samples/csv/orders.csv' EXTRACT FORMAT csv NUM_HEADER_LINES 1 INTO public.orders SELECT $1 as id, $2 as user_id, $3 as product_id, $4 as subtotal, $5 as tax, $6 as total, $7 as discount, $8 as created_at, $9 as quantity, METADATA('filename') AS filename_column; ``` **Load Hive-Style Partitioned Data** This example uses the `CREATE PIPELINE` SQL statement to load data and the value of the search string `'year'` of the filename with each record in a pipeline. The S3 FILTER option contains the filter for the year in a file path. In turn, the `METADATA` function uses the filter `'year'` value to load all records from files with the year `2019` in the specified path. For details about this pipeline definition and setup, see [Data Pipeline Load of CSV Data from S3](/data-pipeline-load-of-csv-data-from-s3). The example assumes the filename follows Hive-style naming standards. ```sql SQL theme={null} CREATE PIPELINE orders_pipeline SOURCE S3 ENDPOINT 'https://s3.us-east-1.amazonaws.com' BUCKET 'ocient-docs' FILTER '/year=2019/**' EXTRACT FORMAT csv NUM_HEADER_LINES 1 INTO public.orders SELECT $1 as id, $2 as user_id, $3 as product_id, $4 as subtotal, $5 as tax, $6 as total, $7 as discount, $8 as created_at, $9 as quantity, METADATA('hive_partition','year') AS hive; ``` ## Related Links [Load Data](/load-data) [CREATE PIPELINE](/data-pipelines#create-pipeline) [Transform Data in Data Pipelines](/transform-data-in-data-pipelines) # Loading Characteristics and Concepts Source: https://docs.ocient.com/loading-characteristics-and-concepts Core concepts behind data loading in Ocient, including parallelism, batching, deduplication, time-key partitioning, and end-to-end throughput considerations. When considering trillions or quadrillions of distinct records being stored and accessed as part of an ever-growing data warehouse, the manner in which data is loaded is important. The loading infrastructure of the data warehouse must not only provide strong correctness and durability guarantees in service of modern data pipelines, but must do so at very high load rates as well. For example, if an individual record averages 1000 bytes in size and the data warehouse must store 100 trillion records, the size of the records to be loaded is 10^14 \* 10^3 = 10^17 bytes, or 100 petabytes. Note that this is *pre-compression* because this is the volume of data that will be presented, row by row, to the data warehouse. If this information represents 1 year’s worth of records, the loading system must be prepared to handle a steady-state average of \~25 Gbps. When the data rate is not completely uniform in time (perhaps displaying cyclic behavior associated with the day/night cycle) it is easy to imagine the required load rate can measure in the 100’s, or even 1000’s of Gbps. Obviously, the sizes and rates are arbitrary, but they are not unrealistic. Consequently, a high-throughput, scalable, and robust loading infrastructure is a first-class citizen in an on-par complexity-wise with the execution engine itself. ## Record-Level Resolution The distinction between "stream" and "batch" is somewhat arbitrary, depending on the system in consideration, but in the loading infrastructure the fundamental unit of loading is a single record in a table. Put another way, the various guarantees listed here are made on a record-by-record basis and not on batches of records. Supporting this is the concept of a ***record stream*** of which the loading infrastructure can manage an unbounded number. A record stream is an ordered sequence of records, and loading considers each record in a stream independently of each other. ## Low Latency Querying A key design goal of the loading system was a low ***time to query-ability*** or ***TTQ***. Strict guarantees are not made but this usually means that any record presented to the loading system is available to be queried on the order of seconds. An interesting property of the implementation is that the variance on TTQ *decreases* as the aggregate record rate *increases.* This is because the loading system maintains internal structures named ***pages*** that accumulate records, and the faster they fill the more quickly they are made available for querying. ## Low Latency Durability A necessary requirement of low-latency querying is the ability to ensure that any particular record is stored in a fault-tolerant manner. To guarantee correctness of query results it must be the case that when a record is considered during query execution (either included or filtered out) it must be considered for *all* subsequent queries until it is deleted by the user. The ***durability*** guarantee made by the loading system must come before query-ability (and therefore is measured in seconds on a record-by-record basis) and matches the system’s configured fault tolerance. For example, if the erasure coding scheme in use has a `K` value of 2, then the loading system must store all records in a manner that allows for at least 2 losses before they can be made queryable. The unit of durability in the loading system is named a ***page*** and within the loading and storage system, a page shares many characteristics with the segments: each page is a self-contained set of records and their own local metadata and statistics, and the consensus-based ownership scheme mediated by the storage clusters is the same for both. However, there are some key distinctions: 1. **Typical Size** — A page is usually substantially smaller than a segment. While segments can be multiple gigabytes in size, pages are more commonly around 100 megabytes. 2. **Replication** — Segments are grouped together into segment groups and mutually erasure coded for fault tolerance. Pages, however, are *replicated* within the storage layer. This is less space efficient for the same fault tolerance, but pages are short-lived: they exist only long enough to be converted into segments, which are denser and richer in metadata. Because they do not exist for very long, the extra storage overhead of the replication scheme is inconsequential, and the loading system opts not to waste the computational effort computing the erasure-coded parity data. Loading page to segment design ## Exactly-Once Guarantee When the record stream sources can support it, the Ocient loading infrastructure is capable of providing exactly-once semantics for every record loaded. This means that the record source and the query layer can be guaranteed that every record presented for load is not only not lost (no "gaps" exist) but also not duplicated. On the surface, this seems like an obvious and simple requirement, but at millions of records per second in a fault-tolerant distributed system, it is extremely non-trivial to provide this guarantee. The loading systems of other database engines can also provide this guarantee, but often with caveats that might not be acceptable. For example, executing a sequence of `INSERT INTO` statements within the context of a transaction is a possible approach, but distributed transaction processing does not easily scale to millions of records per second. Beyond that, there is the issue of how to manage inter-transaction guarantees that place additional requirements on the data sources. If transactions are not in use, it is generally not possible to provide exactly-once guarantees without a bi-directional contract executed by the record source and the loading system. In a very common form of this contract, the loading system provides the exactly-once guarantee by guaranteeing it will de-duplicate any records presented more than once (i.e., the loading system is *idempotent* with respect to record insertions). Then, the source can safely re-send any records the loading system does not indicate it has received. So long as the source resends anything it has not been explicitly told was received (and made durable) the two endpoints can together guarantee every record will be present exactly one time. When this approach is taken, the loading system’s job becomes simple deduplication. On some systems, this is achieved using unique record identifiers. First, the source must be able to indicate a column or set of columns whose values taken together for any particular record uniquely identify it. Then, for each record inserted, a hash set or other structure is consulted. If the unique identifier is not present, it is inserted, if it is, the record is silently dropped. While technically cheap to implement, this scheme suffers because the set structure cannot be allowed to grow indefinitely, and therefore some time or size based bound is placed on the exactly-once guarantee. An additional significant issue is that it is challenging and expensive to ensure the set structure is replicated or distributed amongst all nodes participating in the loading system. This presents correctness and scalability challenges to the exactly-once guarantee. The Ocient loading system takes the idempotent deduplication approach, but performs its deduplication checks in a different manner. The crux of the scheme is that most systems that aim for exactly-once guarantees are also able to replay their record streams *in the same order*. This is true for queuing systems such as as well as file sources such as files stored in S3. Due to this property, the Ocient deduplication scheme tracks a ***durability*** ***horizon*** for each distinct record stream. Recall that a record stream is an *ordered* sequence of records. The durability horizon is an ever-advancing offset into a stream that indicates the record at which the data warehouse guarantees all previous records up to and including that record, for that stream, are guaranteed durable. Using this scheme, the source can easily identify the set of records it should send or re-send. In the loading system, deduplication becomes cheap in both time and space: the system need only store the durability horizon per stream (instead of an unbounded set), and deduplication checks are as simple as a basic interval check comparing whether or not sequences of records stored in pages overlap. ## Linear Scalability at High Rates In order to enable the extremely high record and bit rates associated with hyperscale data pipelines, it is important that the loading system be capable of scaling linearly to address arbitrary loading complexity. The Ocient loading layer achieves true linear scalability due to a shared-nothing approach between nodes executing loading operations. Being linearly scalable, it is relatively straightforward to design a fault-tolerant deployment that can handle any arbitrary rate by simply multiplying by the capabilities of a single node for a specified schema. With that said, it is almost impossible to specify an exact achievable load rate on a per-node basis in advance. This is because the process is bound by the computational complexity of calculating indexes and metadata (which are schema-specified), the width of the erasure coding scheme (which are user-specified), and the bit rate being handled (which is both schema *and* data-dependent). With that said, each Loader Node is usually bounded by memory bandwidth, and it is not uncommon to be capable of steady-state load rates per node measuring in dozens of gigabits per second or millions of records per second. Usually, nodes are also capable of bursting to higher rates: 10’s of millions of records per second and approaching 100 Gbps. ## Conclusion The architecture is designed to maximize performance on the world’s largest data sets. The massively parallel features storage and query processing optimizations to achieve scalable performance on commodity hardware with NVMe SSD storage. The custom-built, fully-integrated SQL execution engine and optimizer deeply integrate with the storage and the input and output layer. The result is a state-of-the-art data warehouse that can deliver outstanding performance loading and querying record sets numbering in the trillions, quadrillions, and beyond. ## Related Links [Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference) [Key Concepts](/key-concepts) # Log Monitoring Source: https://docs.ocient.com/log-monitoring Monitor logs in an Ocient System to detect errors, audit access, and diagnose performance problems across SQL Nodes, Foundation Nodes, and data pipelines. Contact Support for assistance before altering the system logging configuration in the `rolehostd.conf` file. The logs events using these severities: * `Error` * `Warn` * `Info` * `Verbose` * `Debug` * `Extended Debug` By default, the data warehouse creates the text log file `/var/opt/ocient/rolehostd.log` and logs events at the `Info` level and above. Database logging supports extensive customization and integration with third-party tools. You can customize using the `rolehostd` config file. The database can return events in text, JSON, or Extended Log Format (GELF) formats, or the database can return events in a file, to standard output (`stdout`), or to a UDP port. You can customize based on the log source. ## Log Formats The default log format is text. These are some example log entries. ```Text Text theme={null} [2020-06-09T18:11:25.701594][ INFO][role001] roleHost: All roles started [2020-06-09T18:11:25.701881][ INFO][role001] roleHost: Starting all endpoints [2020-06-09T18:11:25.701925][ INFO][role001] roleHost: Starting endpoint 127.0.0.1:17900 [2020-06-09T18:11:25.704905][ INFO][role001] roleHost: Starting endpoint 127.0.0.1:17600 [2020-06-09T18:11:25.708437][ INFO][tcpU008] roleHost: All endpoints started [2020-06-09T18:11:25.709236][ INFO][role001] host: ------------------------------ [2020-06-09T18:11:25.709286][ INFO][role001] host: Rolehostd successfully started [2020-06-09T18:11:25.709305][ INFO][role001] host: ------------------------------ ``` Each log line contains the time, severity, thread, and the source of the log (for example, `roleHost` or `host`). JSON-formatted logs contain the same information but in a structured format. ```json JSON theme={null} {"sys":"oc1.admin0", "timestamp":1591726285.701594684, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"All roles started"} {"sys":"oc1.admin0", "timestamp":1591726285.701881144, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"Starting all endpoints"} {"sys":"oc1.admin0", "timestamp":1591726285.701925519, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"Starting endpoint 127.0.0.1:17900"} {"sys":"oc1.admin0", "timestamp":1591726285.704905192, "thread":"role001", "lvl":" INFO", "src":"roleHost", "msg":"Starting endpoint 127.0.0.1:17600"} {"sys":"oc1.admin0", "timestamp":1591726285.708437413, "thread":"tcpU008", "lvl":" INFO", "src":"roleHost", "msg":"All endpoints started"} {"sys":"oc1.admin0", "timestamp":1591726285.709236006, "thread":"role001", "lvl":" INFO", "src":"host", "msg":"------------------------------"} {"sys":"oc1.admin0", "timestamp":1591726285.709286683, "thread":"role001", "lvl":" INFO", "src":"host", "msg":"Rolehostd successfully started"} {"sys":"oc1.admin0", "timestamp":1591726285.709305246, "thread":"role001", "lvl":" INFO", "src":"host", "msg":"------------------------------"} ``` GELF-formatted log entries follow the format defined on the [GELF via UDP](https://go2docs.graylog.org/5-0/getting_in_log_data/gelf.html) page. This is an example GELF format. ```json JSON theme={null} {"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.701594684, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"All roles started"} {"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.701881144, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"Starting all endpoints"} {"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.701925519, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"Starting endpoint 127.0.0.1:17900"} {"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.704905192, "_thread":"role001", "level":"5", "_source":"roleHost", "short_message":"Starting endpoint 127.0.0.1:17600"} {"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.708437413, "_thread":"tcpU008", "level":"5", "_source":"roleHost", "short_message":"All endpoints started"} {"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.709236006, "_thread":"role001", "level":"5", "_source":"host", "short_message":"------------------------------"} {"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.709286683, "_thread":"role001", "level":"5", "_source":"host", "short_message":"Rolehostd successfully started"} {"host":"oc1.admin0", "version": "1.1", "timestamp":1591726285.709305246, "_thread":"role001", "level":"5", "_source":"host", "short_message":"------------------------------"} ``` The database always returns message content in the `short_message` field of the GELF format. This table maps Ocient severities to GELF system log (syslog) severities. | **Ocient Severity** | **Syslog Severity** | **Syslog Numeric Value** | | ---------------------------- | ------------------- | ------------------------ | | `Error` | `Error` | 3 | | `Warn` | `Warning` | 4 | | `Info` | `Notice` | 5 | | `Verbose` | `Info` | 6 | | `Debug` and `Extended Debug` | `Debug` | 7 | ## Log Appenders A log appender is the destination for log entries. The database can send log entries to a file, a UDP port, or to `stdout`. When logs are sent to the file appender, they grow continuously unless you use the `truncate` option in the configuration. You can use tools such as `logrotate` to manage log files. This is a sample `logrotate` configuration. ```Text Text theme={null} /var/opt/ocient/log/rolehostd.log { rotate 5 daily delaycompress compress copytruncate missingok minsize 1M maxsize 100M } ``` Test the UDP appender using the `netcat` command. ```linux LINUX theme={null} $ netcat -ul 9000 ``` ## Log Configuration An Ocient System supports various logging settings to meet different needs. Set log configuration settings in `rolehostd.conf`, a YAML file normally found in the system `/var/opt/ocient/` directory. You can change configuration settings to suit your system logging needs in this file. For information on checking the status of nodes, see [Statistics Monitoring](/statistics-monitoring). If you are using an Ocient Simulator, logging behavior and directories might differ from these descriptions. For details on how logging works in the Ocient Simulator, see [Simulator Logging](/ocient-simulator#simulator-logging). `rolehostd` **Example** This is an abbreviated `rolehostd.conf` example in YAML format. Normally, this file contains other system configurations, but this tutorial focuses on the `loggingConfig` section, which defines logging behaviors and scopes. ```yaml YAML theme={null} loggingConfig:   '':  # Default scope (all logs)     appender:       logFilePath: /path/to/rolehostd.log    appenders:    - appender:        allowLevel: warn        logFilePath: /path/to/error.log    - appender:        allowLevel: warn        format: json        logFilePath: /path/to/error.json     dump:  # Dump-specific logs     allowLevel: debug     appender:       logFilePath: /path/to/dump.log     query:  # Query-specific logs     allowLevel: info     appender:       format: json       logFilePath: /path/to/query.json    security:  # Security-specific logs     allowLevel: info     appender:       format: json       logFilePath: /path/to/security.json   network-logs:  # Example network logging     allowLevel: info     appender:       type: UDP       server: graylog.internal       port: 12201       format: gelf       compressed: true ``` ### **Scope Definition** The top level of the logging configuration is the scope, which determines the corresponding log sources for each configuration setup. **Default Scope** An empty scope string (`''`) applies its configuration settings to all log entries the system returns. **Named Scopes** Named scopes apply to log entries where the first component matches the name. Useful scopes include: * `query` — Contains information on all queries and statements executed on the system, including users, time, and other details. * `security` — Contains information on all connections made to the system, successful or unsuccessful. * `dump` — Contains information on system failures. ### `allowLevel` The `allowLevel` parameter specifies which severity levels of logs to capture. ```yaml YAML theme={null} allowLevel: value ``` Severity `value` definitions are: * `debug` — Most verbose (includes all logs).  * `info` — Includes informational messages and greater severities.  * `verbose` — More verbose than the `info` severity.   * `warn` — Includes warnings and errors only.  * `error` — Includes only error messages.  ### `appender` Appenders define the format and directory where the database sends log files. You can configure either one or many appenders for each scope. **Single Appender** For a single appender, use the `appender` key. ```yaml YAML theme={null} appender: key: value [ ... ] ``` **Multiple Appenders** For multiple output destinations, use the `appenders` key, followed by a list of `appender` keys. ```yaml YAML theme={null} appenders: - appender: key: value     [ ... ] - appender: key: value     [ ... ] [ -...] ``` ### `format` Defines the output format of the log files of an appender. ```yaml YAML theme={null} appender: format: format_type ``` Supported `format_type` values are: * `text` — Human-readable text format (default).  * `json` — Structured JSON format. * `gelf` — GELF format for integration with Graylog. ### `logFilePath` Defines the destination directory for log files of the appender. The `directory` value is a file path for the appender log files, for example: `/path/to/rolehostd.log`. ```yaml YAML theme={null} appender: logFilePath: directory ``` ### `type` Determines whether the system submits error logs through `stdout` or to a separate server using a TCP or UDP protocol. ```yaml YAML theme={null} appender: # standard output example type: stdout appender: # UDP example type: UDP server: graylog-server.example.com port: 12201 chunkSize: 1024 # Optional compressed: true # Optional appender: # TCP example type: TCP server: 192.158.1.38 port: 12201 ``` `server` Required for TCP or UDP types. The destination DNS name or IP address where the database sends log entries. ```yaml YAML theme={null} appender: server: 192.158.1.38 ``` `port` Required for TCP or UDP types. The port where the database sends log entries. The default value is `9000`. ```yaml YAML theme={null} appender: port: 12201 ``` `chunkSize` Optional for UDP type. The largest packet size in bytes that the system sends to the server. The database discards log entries larger than this value. ```yaml YAML theme={null} appender: chunkSize: 1024 ``` `compressed` Optional for UDP type. If you set this option to `true`, the system compresses packets. For details about log compression, see the [GELF documentation](https://go2docs.graylog.org/current/getting_in_log_data/gelf.html). Otherwise, the packets are uncompressed. ```yaml YAML theme={null} appender: compressed: true ``` ### `truncate` If you set this option to `true`, the system truncates the log file when the system starts. Otherwise, the system appends new logs to the file. The default value is `false`. ```yaml YAML theme={null} appender: truncate: true ``` ### `redactions` You can redact data in logs at the SQL statement level using the optional `redactions` configuration. Enabling redaction creates another version of the logs with redacted SQL text. The `redactions` configuration value is `statement` for the redaction of part of the statement. ```yaml YAML theme={null} appender: redactions: - statement # Optional ``` `rolehostd` **Log Redaction Example** This abbreviated `rolehostd.conf` file configuration enables log redaction at the statement level for both the `rolehostd.log` and `query.json` log files. ```yaml YAML theme={null} loggingConfig: ? '': appenders: - appender: logFilePath: /var/opt/ocient/logs/rolehostd.log - appender: logFilePath: /var/opt/ocient/logs/rolehostd_redacted.log redactions: - statement query: allowLevel: info appenders: - appender: format: json logFilePath: /var/opt/ocient/logs/query.json - appender: format: json logFilePath: /var/opt/ocient/logs/query_redacted.json redactions: - statement ``` ## Related Links [Install an Ocient System](/install-an-ocient-system) [Set Up System Monitoring with the TIG Stack and Kapacitor](/set-up-system-monitoring-with-the-tig-stack-and-kapacitor) *** *Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.* # Looker Connector Source: https://docs.ocient.com/looker-connector Connect Looker Studio to an Ocient System using the Google Apps Script community connector to visualize warehouse data in interactive dashboards and reports. is a free, online data visualization and business intelligence tool for connection to various data sources and creation of interactive dashboards and reports. This connector enables clients to fetch data from an System for visualization in Looker Studio using the [Google Apps Script](https://script.google.com/) platform. ## Prerequisites To use Looker Studio with the Ocient System, you must have this software: * Ocient System — Use the latest version. * A account with login credentials. ## Connect to Ocient Follow these steps to connect Looker Studio to your Ocient System. Open Looker Studio and start a data source. * Go to [https://lookerstudio.google.com/](https://lookerstudio.google.com/) and sign in. * On the home page, select: * **Create** > **Data source**, or * From an open report, **Resource** > **Manage added data sources** > **Add a data source**. Select the Ocient Community Connector. * In the connector gallery, go to the **Partner Connectors** section. * Find and select the **Ocient DB** connector. Looker Studio opens the **Ocient Connector** configuration page in the same tab. Complete the configuration page by entering text in the fields. On the configuration page, select the **Allow to be modified** checkbox next to each field to modify it. This table contains descriptions for each field. | **Field** | **Description** | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Hostname** | Hostname or IP address of the Ocient SQL Node. | | **Database** | Database name in the Ocient System. | | **User** | Ocient username. | | **Password** | Password for the Ocient username. | | **Table** | Ocient table name (`schema.table`). | | **Row Limit** | `LIMIT` value to add to queries. | | **Date Range Column Name** | Column name for the `DATE` or `TIMESTAMP` column in the `WHERE` clause. | | **Sort Column** | Column name for the `ORDER BY` clause. | | **Sort Order** | Sort order with the option of `ASCENDING` or `DESCENDING`. | | **Unnest Option** | Specifies if array columns are unnested. | | **Custom Query** | A query that overrides the query generated by the connector. The Looker data source panel shows only the columns included in this query.

In this mode, Looker Studio infers column data types from the sample output, and not a `DESCRIBE TABLE` SQL statement. | * Select **Connect**. * Select **Allow** when Looker Studio prompts you with "**Allow parameter sharing?**. You can now use Looker Studio to create reports for your Ocient System. Looker Studio uses autocomplete in its filter tool, which can impact date or time filtering for records older than 28 days. To fix this issue, navigate to the Looker filter control and set **Show suggested values** to `off`. ## Ocient Data Types in Looker Studio The connector represents Ocient data types using the closest approximation supported by Looker Studio. The inference of data types happens in Looker Studio in these ways: * In normal mode, the connector generates data types from the table schema produced by a `DESCRIBE TABLE` SQL statement. * In custom query mode, Looker Studio infers data types from sample output. This table shows how Ocient data types correspond to Looker Studio data types. | **Ocient Data Type** | **Looker Studio Data Type** | **Limitations** | | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BOOLEAN | BOOLEAN | None | | TINYINT,
SMALLINT,
INT,
BIGINT,
DOUBLE,
FLOAT,
DECIMAL | NUMBER | None | | CHAR
VARCHAR
BINARY
HASH
IPV4
IP
UUID
VARBINARY
TIME | TEXT | None | | POINT
LINESTRING
POLYGON | TEXT | Looker Studio Community Connectors can use latitude or longitude coordinates, but they do not support native WKT geospatial types. | | DATE | YEAR\_MONTH\_DAY | None | | TIMESTAMP | YEAR\_MONTH\_DAY\_SECOND | None | | CHAR\[],
VARCHAR\[],
BINARY\[],
HASH\[],
IPV4\[],
IP\[],
UUID\[],
VARBINARY\[],
TUPLE\[] | TEXT | Looker Studio does not support arrays or tuples as a native data type. The connector represents container data types as textual strings.

For arrays, use the **Unnest Option** configuration field to flatten array columns for analytics.

For tuples, use the **Custom Query** field to extract the tuple elements. Setting the **Unnest Option** field does not work for flattening tuples. | ## Related Links [Connect to Ocient](/connect-to-ocient) [Ocient Integrations](/ocient-integrations) # Machine Learning in Ocient Source: https://docs.ocient.com/machine-learning-in-ocient Integrate machine learning models into Ocient queries with OcientML for advanced data analysis, empowering predictive and classification insights. The has functionality that enables machine learning training and model execution within the database. Whether you are training a model or using machine learning functionality to do scoring or prediction, OcientML enables machine learning in the SQL syntax directly. You can use the `CREATE MLMODEL` SQL statement to create models. This step is for model training and is referred to as model creation. Similarly, you can use the model on new input data to generate new predictions by executing the scalar function that has the same name as the model. This action is typically referred to as executing the model. Machine learning in depends upon linear algebra functionality that is built into the Ocient System. OcientML provides machine learning capabilities that you can invoke in SQL statements with some application logic. ## Linear Algebra in Ocient matrices are a first-class data type in the Ocient System. Create a matrix using a simple SQL SELECT statement. ```sql SQL theme={null} SELECT {{1, 2}, {3, 4}}; {{1,2},{3,4}} -------------------------------------------------------------------------------- [[1.0, 2.0], [3.0, 4.0]] Fetched 1 row SELECT {{c1*1, c1*2}, {c1*3, c1*4}} FROM sys.dummy2; make_matrix_2x2((2), (2), ((1))*(c1), ((2))*(c1), ((3))*(c1), ((4))*(c1)) -------------------------------------------------------------------------------- [[1.0, 2.0], [3.0, 4.0]] [[2.0, 4.0], [6.0, 8.0]] Fetched 2 rows ``` You can also use shorthand notations to create row `_r` or column vectors `_c`. For example, `_r{1,2,3}` creates a row vector with values `1.0`, `2.0`, and `3.0`. ```sql SQL theme={null} SELECT _r{1,2,3}; _r{1,2,3} -------------------------------------------------------------------------------- [[1.0, 2.0, 3.0]] Fetched 1 row SELECT _c{1,2,3}; _c{1,2,3} -------------------------------------------------------------------------------- [[1.0], [2.0], [3.0]] Fetched 1 row ``` You can execute functions using the values in the vectors. This query does some matrix arithmetic, finds the inverse matrix, and then returns the two eigenvalues and eigenvectors of the inverse. ```sql SQL theme={null} SELECT EIGEN(INVERSE(2 * {{1,2},{3,4}} + {{5,6},{7,8}} / 2)); EIGEN(INVERSE((((2))*({{1,2},{3,4}}))+(({{5,6},{7,8}})/((2))))) -------------------------------------------------------------------------------- [<<-1.3780529228406495, [[0.8013353799887076, -0.5982153531783962]]>>, <<0.05805292284064968, [[0.48196559267508465, 0.8761901434491]]>>] Fetched 1 row ``` ## Machine Learning Models Ocient supports regression, classification, and clustering models. Also, Ocient supports models for dimensionality reduction. For guides on using the different OcientML models, see: * [Regression Analysis](/regression-analysis) * [Classification Analysis](/classification-analysis) * [Clustering Analysis and Dimensionality Reduction](/clustering-analysis-and-dimensionality-reduction) ## Related Links [Machine Learning Model Functions](/machine-learning-model-functions) [Regression Models](/regression-models) # Machine Learning Model Functions Source: https://docs.ocient.com/machine-learning-model-functions Reference for OcientML model functions used in SQL queries to invoke trained machine learning models for prediction, scoring, and feature engineering tasks. functionality supports these SQL functions for machine learning models. The scope of a machine learning model is the schema. ## Supported Machine Learning Models Supported machine learning models and reference material are divided into these categories. For functions to help organize data before training a model, see [Data Preparation](/data-preparation). #### Regression Models * [Simple Linear Regression](/regression-models#simple-linear-regression) * [Multiple Linear Regression](/regression-models#multiple-linear-regression) * [Vector Autoregression](/regression-models#vector-autoregression) * [Polynomial Regression](/regression-models#polynomial-regression) * [Linear Combination Regression](/regression-models#linear-combination-regression) * [Nonlinear Regression](/regression-models#nonlinear-regression) * [Gradient Boosted Trees](/regression-models#gradient-boosted-trees) * [Regression Tree](/regression-models#regression-tree) #### Classification Models * [K Nearest Neighbor Classification](/classification-models#k-nearest-neighbors-classification) * [Naive Bayes Classification](/classification-models#naive-bayes-classification) * [Decision Tree](/classification-models#decision-tree) * [Random Forest](/classification-models#random-forest) * [Logistic Regression](/classification-models#logistic-regression) * [Support Vector Machine](/classification-models#support-vector-machine) * [Gradient Boosted Trees](/classification-models#execute-the-model-6) #### Clustering and Dimension Reduction Models * [Principal Component Analysis](/clustering-and-dimension-reduction-models#principal-component-analysis) * [K-Means Clustering](/clustering-and-dimension-reduction-models#k-means-clustering) * [Gaussian Mixture](/clustering-and-dimension-reduction-models#gaussian-mixture) * [Linear Discriminant Analysis](/clustering-and-dimension-reduction-models#linear-discriminant-analysis) #### Ensemble Models * [Bagging](/ensemble-models#bagging) * [Boosting](/ensemble-models#boosting) * [Stacking](/ensemble-models#stacking) #### Other Models * [Association Rules](/other-models#association-rules) * [Feedforward Neural Network](/other-models#feedforward-neural-network) For a view of the full list of model options, see [Machine Learning Model Options](/machine-learning-model-options). ## Execute a Query Using a Machine Learning Model To create a machine learning model and manage the model, see [Machine Learning Models](/machine-learning-models) for the corresponding syntax. After you create the model, you can execute a query using the model with this syntax. **Syntax** ```sql SQL theme={null} SELECT model_name ( expression [, ... ] ) FROM table_reference ``` | **Parameter** | **Data Type** | **Description** | | ----------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model_name` | identifier | The name of a machine learning model created using a `CREATE MODEL` SQL statement. The model name must be a valid identifier and reference an existing trained model. | | `expression` | identifier
string
numeric
| One or more expressions that serve as input features for the machine learning model evaluation. These expressions must match the expected input schema the model was trained on.

Expressions can be any combination of literal values, column names, arithmetic expressions, and function invocations, with parentheses for grouping. | | `table_reference` | identifier | The name of a table, view, or subquery that provides the data for model evaluation. The table must contain columns or computed expressions that match the expected input features of the model. | **Example** Create a table with data for the model. ```sql SQL theme={null} CREATE TABLE mldemo.mlr AS (SELECT a.c1 AS x1, b.c1 AS x2, 1 + 2*a.c1 + 3*b.c1 AS y FROM sys.dummy10 a, sys.dummy10 b); Modified 100 rows ``` Create a multiple linear regression model based on the data. ```sql SQL theme={null} CREATE MLMODEL mlr_model TYPE MULTIPLE LINEAR REGRESSION ON (SELECT * FROM mldemo.mlr) options('metrics' -> 'true'); Modified 0 rows ``` Execute a `SELECT` query against the multiple linear regression to see the actual and predicted values. Limit the result set to 10 rows. ```sql SQL theme={null} SELECT x1, x2, y AS actual, mlr_model(x1, x2) AS predicted FROM mldemo.mlr LIMIT 10; ``` *Output* ```none Text theme={null} x1 x2 actual predicted ---------------------------------------------------------------- 6 1 16 15.999999999999975 6 2 19 18.999999999999975 6 3 22 21.999999999999975 6 4 25 24.999999999999975 6 5 28 27.999999999999975 6 6 31 30.999999999999975 6 7 34 33.99999999999997 6 8 37 36.99999999999997 6 9 40 39.99999999999997 6 10 43 42.99999999999997 Fetched 10 rows ``` ## Related Links [Machine Learning in Ocient](/machine-learning-in-ocient) [Machine Learning Models](/machine-learning-models) # Machine Learning Model Options Source: https://docs.ocient.com/machine-learning-model-options Reference for OcientML model options, including required and optional parameters for association rules, bagging, boosting, decision trees, k-means, and more. This list of options contains model options for all models in the System. ## Association Rules ### Model Options #### Optional `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. ## Bagging ### Model Options #### Required `baseModels` — This option specifies the children of the bagging model. You must specify this value as a JSON array, where each object in the array has the three fields `type`, `count`, and `options`. `taskType` — This option specifies the type of task, which must be either `CLASSIFICATION` or `REGRESSION` depending on the type of model for training. #### Optional `ROCNumSamples` — If you set this option, you must specify a positive integer that represents the number of samples for the model to use when calculating the area under the ROC curve. You must also set the `metrics` option to `true`. The default value is the number of child models. `bootstrap` — If you set this option to `true`, the model uses bootstrap sampling with replacement, meaning each child model trains on a random subset of the data (either the `rowsPerChild` or `fractionSelected` value sets the exact number of rows), and the same row can appear multiple times for each child. If you set this option to `false`, the model does not use replacement, meaning each row can appear at most one time per child. The default value is `false`. `continuousFeatures` — If you set this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start with 1. In the default state, the model considers no features as continuous. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `fractionSelected` — If you set this option, the option represents the proportion of rows the model uses to train each child model. The value is a `double` that must be in the interval (0, 1]. You cannot set this option if you also set the `rowsPerChild` option to a positive value. The default behavior is that the model uses all available rows. `inputsPerChild` — If you set this option, the option represents the number of features used to create each child model. The default value is the total number of features divided by 3 and rounded up. `maxChildThreads` — If you set this option, the value must be an integer representing the maximum number of threads each child model can use. If a child accepts a `maxThreads` option, the model passes this value to the child. `maxThreads` — If you set this option, the option represents the maximum number of parallel threads to use while the model trains. This value must be a positive integer. The default value is 16. `metrics` — If you set this option to `true`, the system calculates certain metrics depending on the value of the `taskType` option. If you set the `taskType` option to `CLASSIFICATION`, the metrics are the percentage of correctly classified rows and the area under the ROC curve. If you set the value to `REGRESSION`, the metrics are the root mean square error and the adjusted R-squared. The default value is `false`. `noSnapshot` — If you set this option to `true`, the data source must not change. In this case, the database does not create an intermediate table that stores the result of the specified SQL statement, which the model uses for training a random forest. Child decision trees always have this option set to `true`, so the database does not create a separate intermediate table for each decision tree. The default value is `false`. Setting this option to `true` can speed up training when the training set is fixed. `requiredFeatures` — If you set this option, the value must be a comma-separated list of integers representing features starting at index 1. The bagging model passes these features down to every child. The default value is an empty list, meaning there is no required feature. `rowsPerChild` — If you set this option to a positive integer, the number represents the number of rows (from a random sample) to use for each decision tree. If you set this option to 0, each child uses all available rows. The default value is 0. You cannot set this option to a positive value if you also set the `fractionSelected` option. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. `weighted` — If you set this option, the system passes the value directly to child models that support it. The behavior depends on the model type of the target child. ## Boosting ### Model Options #### Required `baseModels` — This option specifies the children of the bagging model. You must specify this as a JSON array, where each object in the array has the three fields `type`, `count`, and `options`. `learningRate` — A `decimal` value between 0.0 and 1.0 that tunes how much the model learns from each successive child. `taskType` — This option specifies the type of task, which must be either `CLASSIFICATION` or `REGRESSION` depending on the type of model for training. #### Optional `ROCNumSamples` — If you set this option, you must specify a positive integer that represents the number of samples for the model to use when calculating the area under the ROC curve. You must also set the `metrics` option to `true`. The default value is `10`. `continuousFeatures` — If you set this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start with 1. In the default state, the model considers no features as continuous. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `fractionSelected` — If you set this option, the option represents the proportion of rows the model uses to train each child model. The value is a `double` that must be in the interval (0, 1]. You cannot set this option if you also set the `rowsPerChild` option to a positive value. The default behavior is that the model uses all available rows. `inputsPerChild` — If you set this option, the value must be an `integer` type that is greater than or equal to 1, which specifies the number of input features each boosting child should use. This value cannot exceed the number of input features available in the data set. When you specify this value, the algorithm deterministically cycles through pre-enumerated feature subsets to ensure each child uses exactly the specified number of features. When you do not specify this value, the model uses all available features for each child. `lossFunction` — If you set this option, the value represents the loss function used by the model. Accepted values are: `'squared_error'` and `'log_loss'`. When you set this value to `'squared_error'`, the model calculates errors as the squared difference between predicted and actual values. The target column must contain numeric values. This is the default value when the `taskType` option is set to `REGRESSION`. When you set this value to `'log_loss'`, the model calculates errors using logistic loss. This is the default value when the `taskType` option is `CLASSIFICATION`. `maxThreads` — If you set this option, the option represents the maximum number of parallel threads to use while the model trains. This value must be a positive integer. The default value is 16. `metrics` — If you set this option to `true`, the system calculates certain metrics depending on the value of the `taskType` option. If the `taskType` option is set to `CLASSIFICATION`, the metrics are the percentage of correctly classified rows and the area under the ROC curve. If the `taskType` option is set to `REGRESSION`, the metrics are the root mean square error and the adjusted R-squared. The default value is `false`. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Decision Tree ### Model Options #### Optional `ROCNumSamples` — If you set this option, you must specify a positive integer that represents the number of samples for the model to use when calculating the area under the ROC curve. You must also set the `metrics` option to `true`. The default value is `10`. `continuousFeatures` — If you set this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start with 1. In the default state, the model considers no features as continuous. `distinctCountLimit` — If you set this option, the value must be a positive `integer` type. This value sets the limit for how many distinct values a non-continuous feature and label can contain. The default value is `256`. `doPrune` — If you set this option to `true`, the model uses Pessimistic Error Pruning (PEP) to prune the tree after training. The default value is `false`. `enableResplits` — If you set this option, it must be a `boolean` type that determines if the tree can reuse the same continuous feature multiple times along a single branch (e.g., split on `x1 < 7` and later `x1 < 3`). This action can capture more complex, range-specific relationships. The default value is `true`, meaning that continuous features remain available for additional splits after use, thereby allowing the tree to create more complex decision boundaries. If you set this option to `false`, the model marks continuous features as exhausted after their first use, and the model cannot use them again in subsequent splits in the same tree. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `featureSubsetStrategy` — If you set this option, the option specifies how many features the decision tree should consider at each split from the still-available features. When this value is higher, the model has a higher accuracy and lower variance, but takes longer to train. You can specify this option either as an integer (e.g., `4`, meaning consider up to four features at each split) or one of these values: `all` (check every feature), `sqrt` (check up to the square root of the number of total features), and `one-third` (check up to one-third of the number of total features). The default value is `all`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `maxCellsToFetch` — If you set this option, the value must be a positive integer. Controls the chunking behavior when fetching feature values during model training. The limit represents the maximum number of data cells (calculated as number of columns × number of rows) that can be fetched in a single operation, not a byte limit. When the expected data size exceeds this threshold, the algorithm switches to database-based processing using SQL queries instead of in-memory processing. This value defaults to 33,554,432 cells (calculated as 32 × 1024 × 1024). `maxDepth` — If you set this option, the value must be a positive integer. This value sets the maximum allowable depth of the decision tree (the maximum number of features to split on). The default is unspecified, which means there is no maximum depth. `maxRows` — If you set this option, the value must be a positive integer. This option limits the number of rows used for model training by creating a snapshot table with only the specified number of rows from the input query. This option cannot be used with `noSnapshot -> true` (attempting to set both results in an `invalid argument` error during model creation). When this option is unspecified, the model trains using all rows from the input query. `maxThreads` — If you set this option, the value must be a positive integer. This value indicates the maximum number of parallel threads to use while the model trains. The default value is 2. `metrics` — If you set this option to `true`, the model also calculates the percentage of samples correctly classified by the model and saves this information in a catalog table. This option defaults to `false`. `noSnapshot` — If you set this option to `true`, the database does not create an intermediate table that stores the result of the specified SQL statement, which the model uses for training. This option defaults to `false`. In this case, the database creates and uses the intermediate table. Setting this option to `true` is useful when the training set is fixed. If the training set is a table with modifications, set this option to `false` as the decision tree trainer uses different data sets in different parts of the tree. Likewise, if the training set consists of a query that returns 100 rows, then set this option to `false` because there is no guarantee that running that query twice generates the same 100 rows each time. `numSplits` — If you set this option, the value must be an integer greater than 1. This value sets the maximum number of binary branches a continuous feature can consider. The default value is 32. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `skipLimitCheck` — If you set this option to `true`, the model skips cardinality checks that throw errors when columns have too many values. The limit that this option checks is the same one specified by the `distinctCountLimit` option. The default value is `false`. `splitMetric` — If you set this option, the option controls which function the model uses to evaluate the quality of a split during tree construction. Supported options are: `gini_impurity` (measures impurity based on class distributions), `macro_f1` (uses the macro-averaged F1 score to guide splits), `micro_f1` (uses the micro-averaged F1 score to guide splits), and `weighted_f1` (uses the class-frequency-weighted F1 score to guide splits). The default value is `gini_impurity`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. `suppressJIT` — If you set this option to `true`, the model suppresses just-in-time code generation. `weighted` — If you set this option, the model considers weights for labels. If you set this option value to `true`, you must specify an additional column as a `double` in the training data for label weights. Rows with the same labels must have the same weights. If you set this value to `auto`, the model calculates weights automatically by weighting each label according to the ratio of the count of the most frequent label to the count of the specified label. As a result, the most frequent label has the weight `1.0` and the other label weights are higher. The default value is `false`, which means all labels have equal weight. ## Feedforward Network ### Model Options #### Required `hiddenLayerSize` — You must set this option to a positive `integer` type that specifies the number of nodes in each hidden layer. `hiddenLayers` — You must set this option to a positive `integer` type that specifies how many hidden layers to use. `lossFunction` — This option specifies the loss function that all hidden layer nodes and all output layer nodes use. This function can be one of several predefined loss functions or a user-defined loss function. The predefined loss functions are `squared_error` (regression), `vector_squared_error` (vector-valued regression), `log_loss` (binary classification with target values of 0 and 1), `logits_loss` (binary classification with target values of 0 and 1), `hinge_loss` (binary classification with target values of -1 and 1), and `cross_entropy_loss` (multi-class classification). If the value for this required option is none of these strings, the model assumes a user-defined loss function. The user-defined loss function specifies the per-sample loss. Then, the actual loss function is the sum of this function applied to all samples. The model should use the variable `y` to refer to the dependent variable in the training data, and the model should use the variable `f` to refer to the computed estimate for the specified sample. `outputs` — You must set this option to a positive integer that specifies the number of outputs. #### Optional `activationFunction` — If you set this option, the values are `linear`, `relu` (rectified linear unit), `leakyrelu` (leaky rectified linear unit), `tanh` (hyperbolic tangent function), or `sigmoid` (fast sigmoid approximation). The default value is `relu`. This option affects all layers except the output layer. `adamBeta1` — If you set this option, the option represents the value of β₁ in the Adam optimization algorithm. For higher values of this option, training is less noisy but takes longer to converge. The default value is `0.9`. `adamBeta2` — If you set this option, the option represents the value of β₂ in the Adam optimization algorithm. For higher values of this option, training is less noisy but takes longer to converge. The default value is `0.99`. `adamEpsilon` — If you set this option, the option represents the value of ε in the Adam optimization algorithm. For higher values of this option, training is more numerically stable but takes longer to converge. The default value is `1e-7`. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `finiteDifferenceH` — If you set this option, the value must be a `double` representing the step size (`h`) for approximating gradients using the finite difference method. The model uses this value only if analytical gradients are not active. This value should generally be a small positive value, typically from `0.0001` (`1e-4`) to `0.0000001` (`1e-7`). The default value is `0.00001` (`1e-5`). `gradientClipThreshold` — If you set this option, the value must be a `double` that represents the gradient norm threshold for clipping. When the overall gradient norm exceeds this threshold, the system scales all gradient components uniformly to preserve direction. This operation prevents issues with exploding gradients in unstable loss landscapes. Set this value to 0 or a negative value to disable gradient clipping. The default value is `1000000` (`1e6`). `learningRate` — If you set this option, the value must be a `double` type representing the base learning rate for the Adam (Adaptive Moment Estimation) machine learning optimizer. Adam adapts this rate individually for each parameter during training. A common starting point for Adam is `0.001` (`1e-3`). Valid values must be positive and are generally in the range of `0.00001` (`1e-5`) to `0.01` (`1e-2`). A higher learning rate can speed up training, but can cause the optimizer to overshoot and miss optimal solutions. Conversely, a lower learning rate ensures more stable and precise convergence but can make training much slower. If you do not specify this option, the system automatically selects a learning rate and adjusts it during training using the 1Cycle learning schedule. Specifying a learning rate disables automatic adjustment and instead uses a fixed learning rate value. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `maxInitParamValue` — If you set this option, the value must be a floating-point number. Sets the maximum for initial parameter values in the optimization algorithm. The default value is `1`. `metrics` — If you set this option to `true`, the model calculates the average value of the loss function. `minInitParamValue` — If you set this option, the value must be a floating-point number. Sets the minimum for initial parameter values in the optimization algorithm. The default value is `-1`. `normalize` — If you set this option to `true`, this option applies z-score normalization to inputs by default, storing means and standard deviation, and automatically applying them at inference. The default value is `true`. `numEpochs` — If you set this option, the value must be a positive `integer` type representing the maximum number of epochs, or full passes, during training through the entire data set. If you do not specify this option, the default maximum is `200`, but training typically stops earlier due to automatic early stopping when the model has converged. `outputActivationFunction` — If you set this option, the values are `linear`, `relu` (rectified linear unit), `leakyrelu` (leaky rectified linear unit), `tanh` (hyperbolic tangent function), or `sigmoid` (fast sigmoid approximation). Different activation functions have different output ranges. The chosen activation function should match the dependent variable of your data. For example, if the dependent variable can be anything, then choose the `linear` value. If the dependent variable is always positive, then choose the `relu` value. If your outputs range from -1 to 1 or you perform hinge loss classification, `tanh` is a good option because the hyperbolic tangent function has the same range. But, if your outputs range from 0 to 1 or you perform log loss classification, `sigmoid` is a better choice for the same reason. This option defaults to `linear`. The option only sets the activation function for the output layer. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `randomSeed` — If you set this option, the value must be a positive `integer` type representing the seed for the random number generator the system uses for weight initialization. Setting this option makes model training deterministic (given the same data and options). If you do not specify this option or set it to `0`, the system uses a non-deterministic random seed. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. `useSoftmax` — If you set this option to `true`, the model applies a softmax function to the output of the output layer before computing the loss function. The default value is `true` if you set the `lossFunction` to `cross_entropy_loss`, and `false` otherwise. ## Gaussian Mixture Model ### Model Options #### Required `numDistributions` — This option must be a positive `integer` type that specifies the number of clusters of Gaussian distributions for the model to make. #### Optional `epsilon` — If you specify this option, the value must be a valid positive floating point number. When the maximum distance that the entire best model moves in its n-dimensional space is less than this value, the algorithm terminates. The default value is `0.00000001` (`1e-8`). `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `maxIterations` — If you set this option, the option represents the maximum number of optimization iterations to train the model. For higher values of this option, the model is likelier to converge to the expected `epsilon`, but it might take longer to train. The default value is `100`. `normalize` — If you set this option to `true`, the model automatically computes the mean and standard deviation of each feature and uses them to normalize the data during training. Defaults to `true`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Gradient Boosted Trees ### Model Options #### Required `learningRate` — A `decimal` value between 0.0 and 1.0 that tunes how much the model learns from each successive child. `numChildren` — An `integer` value representing the total number of trees to build sequentially. Each tree learns to correct the errors of the previous trees. #### Optional `continuousFeatures` — If you set this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start with 1. In the default state, the model considers no features as continuous. `enableResplits` — If you set this option, the value must be a `boolean` type that determines if the tree can reuse the same continuous feature multiple times along a single branch (e.g., split on `x1 < 7` and later `x1 < 3`). This action can capture more complex, range-specific relationships. The default value is `true`, meaning that continuous features remain available for additional splits after use, thereby allowing the tree to create more complex decision boundaries. If you set this option to `false`, the model marks continuous features as exhausted after their first use, and the model cannot use them again in subsequent splits in the same tree. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `fractionSelected` — If you set this option, the option represents the proportion of rows the model uses to train each child model. The value is a `double` that must be in the interval (0, 1]. You cannot set this option if you also set the `rowsPerChild` option to a positive value. The default behavior is that the model uses all available rows. `inputsPerChild` — If you set this option, the value must be an `integer` type greater than or equal to 1 that specifies the number of input features each boosting tree should use. This value cannot exceed the number of input features available in the data set. When you specify this value, the algorithm deterministically cycles through pre-enumerated feature subsets to ensure each tree uses exactly the specified number of features. The default behavior is that the model uses all available features for each tree. `lossFunction` — If you set this option, the option represents the loss function used and determines the type of task the model does. Accepted values are: `'squared_error'` and `'log_loss'`. When you set this value to `'squared_error'`, the model calculates errors as the squared difference between predicted and actual values. The target column must contain numeric values. This is the default value for regression tasks. When you set this value to `'log_loss'`, the model calculates errors using logistic loss. This is the default value for classification tasks. `maxCellsToFetch` — If you set this value, the value must be an `integer` type that determines the memory threshold to switch from training with system memory to training with SQL queries in the database. In-memory training is generally faster, but is limited by the available SQL Node memory. If the size of a training data subset exceeds this value, then the system performs training operations using SQL queries. The default value is `33,554,432` (calculated as `32 * 1024 * 1024`). `maxDepth` — If you set this value, the value must be a positive `integer` type that represents the maximum allowable depth of the child trees. The default value is `3`. `maxThreads` — If you set this value, the value must be a positive `integer` type that sets the maximum number of parallel threads to use for training each child decision tree. Parallel threads do not affect the sequential method of training each tree. The default value is `16`. `metrics` — If you set this value to `true`, the system calculates and stores final model metrics (`R²/RMSE` for regression or `Accuracy/LogLoss` for classification) on the training data. The default value is `false`. `numSplits` — If you set this option, the value must be an `integer` greater than 1. This value sets the maximum number of binary branches a continuous feature can consider. The default value is `32`. `resplitDepth` — If you set this option, the value must be an `integer` type that sets the maximum depth at which tree nodes can be re-split during optimization. This option controls how deep the algorithm searches for better split points. The default value is `6`. `resplitThreshold` — If you set this option, the value must be a `decimal` type that sets the minimum improvement threshold required to trigger a re-split operation. Lower values allow more aggressive re-splitting but can increase training time. Higher values require larger improvements to trigger re-splits. The default value is `0.1`. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Kmeans ### Model Options #### Required `k` — This option must be a positive `integer` type that specifies how many clusters to make. #### Optional `epsilon` — If you specify this option, the value must be a valid positive floating point value. When the maximum distance that a centroid moves from one iteration of the algorithm to the next is less than this value, the algorithm terminates. The default value is `0.00000001` (`1e-8`). `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `lloydRounds` — If you set this option, the option represents the maximum number of iterations of the Lloyd algorithm to train the model after guessing the centroids. For higher values of this option, the model is more likely to be accurate but takes longer to train. The default value is `20`. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `normalize` — If you set this option to `true`, the model normalizes the data before the start of training. The default value is `true`. `oversampling` — If you set this option, the option represents the number of candidate guesses for the model to choose in the parallel-round phase of k-means||. For higher values of this option, the model is more likely to be accurate but takes longer to train. The default value is `k`. `parallelRounds` — If you set this option, the option represents the minimum number of parallel rounds for which the k-means|| algorithm runs. For higher values of this option, the model is more likely to be accurate but takes longer to train. The default value is `8`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## K Nearest Neighbors ### Model Options #### Required `k` — This option must be a positive `integer` type that specifies how many closest points to use for classifying a new point. #### Optional `distance` — If you set this option, the value must be a function in SQL syntax for calculating the distance between a point used for classification and points in the training data set. This function should use the variables x1, x2, …​ for the 1st, 2nd, … features in the training data set, and p1, p2, …​ for the features in the point for classification. The default value is the Euclidian distance function. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `normalize` — If you set this option to `true`, the model automatically computes the mean and standard deviation of each feature and uses them to normalize the data during training. The default value is `true`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. `weight` — If you specify this option, the value must be a function in SQL syntax for calculating the weight of a neighbor. The function should use the variable d for distance. By default, the distance is set to `1.0 / (d + 0.1)`, thus avoiding division by zero on exact inputs and still allowing neighbors to have some influence. ## Linear Combination Regression ### Model Options #### Required `functionN` — You must specify the first function using a key named `'function1'`. Subsequent functions must use keys with names that use subsequent values of N. You must specify functions in SQL syntax and should use the variables `x1, x2, ..., xn` to refer to the 1st, 2nd, and nth independent variables, respectively. For example,`'function1' -> 'sin(x1 * x2 + x3)', 'function2' -> 'cos(x1 * x3)'`. #### Optional `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `gamma` — If you set this option, the value must be a matrix. This value represents a Tikhonov gamma matrix used for regularization. For details, see [Tikhonov regularization](https://en.wikipedia.org/wiki/Tikhonov_regularization). `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `metrics` — If you set this option to `true`, the model collects quality metrics such as the coefficient of determination (R-squared), the adjusted coefficient of determination, and the root mean squared error (RMSE). The default value is `false`. `normalize` — If you set this option to `true`, the model uses auto-scaling to compute the mean and standard deviation of each input feature to normalize data during training, making training more numerically stable. The model then unscales parameters so the persisted model operates in the original units. The default value is `true`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. `threshold` — This option enables soft thresholding. If you specify this option, the option must be a positive numeric value. After the model calculates the coefficients, if any coefficients are greater than the threshold value, the model subtracts the threshold value from the coefficients. If any coefficients are less than the negation of the threshold value, the model adds the threshold value to the coefficients. For any coefficients between the negative and positive threshold values, the model sets the coefficients to zero. `weighted` — If you set this option to `true`, the model performs weighted least squares regression, where each sample has an associated weight or importance. When weighted, there is an extra numeric column after the dependent variable that represents the weight of the sample. The default value is `false`. `yIntercept` — If you set this option, then the option must be a numeric value. The system forces the specific y-intercept (i.e., the model value when `x` is zero). ## Linear Discriminant Analysis ### Model Options #### Optional `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `normalize` — If you set this option to `true`, the model automatically computes the mean and standard deviation of each feature and uses them to normalize the data during training. The default value is `true`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Logistic Regression ### Model Options #### Optional `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `metrics` — If you set this option to `true`, the model calculates the percentage of samples that are correctly classified by the model and saves this information in the `sys.logistic_regression_models` system catalog table. The default value is `false`. `normalize` — If you set this option to `true`, the model uses auto-scaling to compute the mean and standard deviation of each input feature to normalize data during training, making training more numerically stable. The model then unscales parameters so the persisted model operates in the original units. The default value is `true`. `numEpochs` — If you set this option, the value must be a positive `integer` type representing the maximum number of IRLS iterations during training. The default value is `20`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Multiple Linear Regression ### Model Options #### Optional `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `gamma` — If you set this option, the option must be a matrix. The value represents a Tikhonov gamma matrix used for regularization. For details, see [Tikhonov regularization](https://en.wikipedia.org/wiki/Tikhonov_regularization). The model uses this option for ridge regression. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `metrics` — If you set this option to `true`, the model collects quality metrics such as the coefficient of determination (R-squared), the adjusted coefficient of determination, and the root mean squared error (RMSE). The default value is `false`. `normalize` — If you set this option to `true`, the model uses auto-scaling to compute the mean and standard deviation of each input feature to normalize data during training, making training more numerically stable. The model then unscales parameters so the persisted model operates in the original units. The default value is `true`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. `threshold` — If you set this option, the option enables soft thresholding. The value must be a positive number. After the model calculates the coefficients, if any coefficients exceed the threshold value, the model subtracts the threshold value from those coefficients. If any coefficients are less than the negation of the threshold value, the model adds the threshold value to the coefficients. For any coefficients that are between the negative and positive threshold values, the model sets the coefficients to zero. `weighted` — If you set this option to `true`, the model performs weighted least squares regression, where each sample has a weight or importance associated with it. In this case, the table contains an additional numeric column after the dependent variable, which contains the weight for the sample. The default value is `false`. `yIntercept` — If you set this option, then the option must be a numeric value. The system forces the specific y-intercept (i.e., the model value when `x` is zero). ## Naive Bayes ### Model Options #### Optional `continuousFeatures` — If you set this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start with 1. In the default state, the model considers no features as continuous. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `metrics` — If you set this option to `true`, the model calculates the percentage of samples correctly classified by the model and saves this information in a system catalog table. The default value is `false`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Nonlinear Regression ### Model Options #### Required `function` — Specify the name of the function to fit the data in SQL syntax. Use `a1, a2, …` to refer to the parameters for optimization. Use `x1, x2, …` to refer to the input features. The model does not allow some SQL functions. The model allows only scalar expressions that can be represented internally as postfix expressions. Most notably, the model does not allow some functions that are rewritten as CASE statements (like `least()` and `greatest()`). If your function is not allowed, the model displays an error message. `numParameters` — Specify this option as a positive integer. This value specifies the number of different parameters to optimize, i.e., how many different `aN` variables there are in the user-specified function. #### Optional `adamBeta1` — If you set this option, the option represents the value of β₁ in the Adam optimization algorithm. For higher values of this option, training is less noisy but takes longer to converge. The default value is `0.9`. `adamBeta2` — If you set this option, the option represents the value of β₂ in the Adam optimization algorithm. For higher values of this option, training is less noisy but takes longer to converge. The default value is `0.99`. `adamEpsilon` — If you set this option, the option represents the value of ε in the Adam optimization algorithm. For higher values of this option, training is more numerically stable but takes longer to converge. The default value is `1e-7`. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `finiteDifferenceH` — If you set this option, the value must be a `double` type representing the step size (`h`) for approximating gradients using the finite difference method. The model uses this value only if analytical gradients are not active. This value should generally be a small positive number, typically from `0.0001` (`1e-4`) to `0.0000001` (`1e-7`). The default value is `0.00001` (`1e-5`). `gradientClipThreshold` — If you set this option, the value must be a `double` that represents the gradient norm threshold for clipping. When the overall gradient norm exceeds this threshold, the system scales all gradient components uniformly to preserve direction. This operation prevents issues with exploding gradients in unstable loss landscapes. Set this value to 0 or a negative value to disable gradient clipping. The default value is `1000000` (`1e6`). `lassoCoefficient` — If you specify this option, the value must be a `double` data type. This option is the lasso coefficient for the loss function. The default behavior is the function ignores this option, effectively setting this option to `0.0`. `learningRate` — If you set this option, the value must be a `double` type representing the base learning rate for the Adam (Adaptive Moment Estimation) machine learning optimizer. Adam adapts this rate individually for each parameter during training. A common starting point for Adam is `0.001` (`1e-3`). Valid values must be positive and are generally in the range of `0.00001` (`1e-5`) to `0.01` (`1e-2`). A higher learning rate can speed up training, but can cause the optimizer to overshoot and miss optimal solutions. Conversely, a lower learning rate ensures more stable and precise convergence but can make training much slower. If you do not specify this option, the system automatically selects a learning rate and adjusts it during training using the 1Cycle learning schedule. Specifying a learning rate disables automatic adjustment and instead uses a fixed learning rate value. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `lossFunction` — If you set this option, the option indicates to the nonlinear optimizer the loss function to use on a per-sample basis. Then, the actual loss function is the sum of this function applied to all samples. The model should use the variable `y` to refer to the dependent variable in the training data and the variable `f` to refer to the computed estimate for the specified sample. The default is the least squares function, which you can specify as `(f-y)*(f-y)`. `maxInitParamValue` — If you specify this option, the value must be a floating-point number. This option sets the maximum for initial parameter values in the optimization algorithm. The default value is `1`. `metrics` — If you set this option to `true`, the model calculates the coefficient of determination (R-squared), the adjusted R-squared, and the root mean squared error (RMSE). However, the model calculates these quality metrics using the least squares loss function, and not the user-specified loss function, because these metrics only make sense for least squares. The default value is `false` `minInitParamValue` — If you specify this option, the value must be a floating-point number. This option sets the minimum for initial parameter values in the optimization algorithm. The default value is `-1`. `numEpochs` — If you set this option, the value must be a positive `integer` type representing the maximum number of epochs, or full passes, during training through the entire data set. If you do not specify this option, the default maximum value is `200` for Adam optimization or `100` for Levenberg-Marquardt, but training typically stops earlier due to automatic early stopping when the model has converged. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `randomSeed` — If you set this option, the value must be a positive `integer` type representing the seed for the random number generator the system uses for weight initialization. Setting this option makes model training deterministic (given the same data and options). If you do not specify this option or set it to `0`, the system uses a non-deterministic random seed. `ridgeCoefficient` — If you specify this option, the value must be a `double` data type. This option is the ridge coefficient for the loss function. The default behavior is the function ignores this option, effectively setting this option to `0.0`. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Polynomial Regression ### Model Options #### Required `order` — This option is the degree of the polynomial and must be set to a positive integer. #### Optional `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `gamma` — If you specify this option, the value must be a matrix. The value represents a Tikhonov gamma matrix that is used for regularization. For details, see [Tikhonov regularization](https://en.wikipedia.org/wiki/Tikhonov_regularization). `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `metrics` — If you set this option to `true`, the model collects quality metrics such as the coefficient of determination (R-squared), the adjusted coefficient of determination, and the root mean squared error (RMSE). The default value is `false`. `negativePowers` — If you set this option to `true`, the model includes independent variables raised to negative powers. These variables are named Laurent polynomials. The model generates all possible terms such that the sum of the absolute value of the power of each term in each product is less than or equal to the order. For example, with two independent variables and the order set to `2`, the model is: `y = a1*x1^2 + a2*x1^-2 + a3*x2^2 + a4*x2^-2 + a5*x1*x2 + a6*x1^-1*x2 + a7*x1*x2^-1 + a8*x1^-1*x2^-1 + a9*x1 + a10*x1^-1 + a11*x2 + a12*x2^-1 + b`. The default value is `false`. `normalize` — If you set this option to `true`, the model uses auto-scaling to compute the mean and standard deviation of each input feature to normalize data during training, making training more numerically stable. The model then unscales parameters so the persisted model operates in the original units. The default value is `true`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. `threshold` — This option enables soft thresholding. If you specify this option, then the option must be a positive numeric value. After the model calculates the coefficients, if any of them are greater than the threshold value, the threshold value is subtracted from them. If any coefficients are less than the negation of the threshold value, the model adds the threshold value to them. For any coefficients that are between the negative and positive threshold values, the model sets those coefficients to zero. `weighted` — If you set this option to `true`, the model performs weighted least squares regression, where each sample has an associated weight. When weighted, there is an extra numeric column after the dependent variable that has the weight for the sample. The default value is `false`. `yIntercept` — If you set this option, then the option must be a numeric value. The system forces the specific y-intercept (i.e., the model value when `x` is zero). ## Principal Component Analysis ### Model Options #### Optional `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Random Forest ### Model Options #### Required `numChildren` — Number of child decision trees. #### Optional `ROCNumSamples` — If you set the option, you must also set the `metrics` option. This positive integer indicates the number of samples for the model to use for the area under the ROC curve. The default value is the number of child decision trees. `bootstrap` — If you set this option to `true`, the model uses bootstrap sampling with replacement, meaning the model trains each tree in the random forest on a random subset of the data (either the `rowsPerChild` or `fractionSelected` option sets the exact number of rows), and the same row can appear multiple times in each tree. If you set this option to `false`, this option does not use replacement, meaning each row can appear at most once per tree. The default value is `false`. `continuousFeatures` — If you set this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start with 1. In the default state, the model considers no features as continuous. `distinctCountLimit` — If you set this option, the value must be a positive integer. This value limits how many distinct values a non-continuous feature and the label can contain. The default value is `256`. `doPrune` — If you set this option to `true`, the model uses Pessimistic Error Pruning (PEP) to prune the tree after training. The default value is `false`. `enableResplits` — If you set this option, the value must be a `boolean` type that determines if the tree can reuse the same continuous feature multiple times along a single branch (e.g., split on `x1 < 7` and later `x1 < 3`). This action can capture more complex, range-specific relationships. The default value is `true`, meaning that continuous features remain available for additional splits after use, allowing the tree to create more complex decision boundaries. When you set this option to `false`, the model marks continuous features as exhausted after their first use, and the model cannot use them again in subsequent splits in the same tree. The model passes this option directly to the child decision trees. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `featureSubsetStrategy` — If you set this option, the model passes this option directly to the child decision trees. The option specifies how many features the child decision trees should consider at each split from the still-available features. When this value is higher, the model will have higher accuracy and lower variance, but takes longer to train. You can specify this option either as an integer (e.g., `4`, meaning consider up to 4 features at each split) or one of the three possible string options: `all` (checks every feature), `sqrt` (checks up to the square root of the number of total features), and `one-third` (checks up to one-third of the number of total features). The default value is `all`. `fractionSelected` — If you set this option, the option represents the proportion of rows the model uses to train each child model. The value is a `double` that must be in the interval (0, 1]. You cannot set this option if you also set the `rowsPerChild` option to a positive value. The default behavior is that the model uses all available rows. `inputsPerChild` — If you set this option, the option specifies the number of features for the creation of each child decision tree. The default value is the number of features you specify for the forest divided by 3 and rounded up. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `maxCellsToFetch` — If you set this option, the model passes this option directly to the child decision trees. If you set this option, the value must be a positive integer. Controls the chunking behavior when fetching feature values during model training. The limit represents the maximum number of data cells (calculated as the number of columns × number of rows) that the system can fetch in a single operation, not a byte limit. When the expected data size exceeds this threshold, the algorithm switches to database-based processing using SQL queries instead of in-memory processing. The default value is `33,554,432` cells (calculated as `32 × 1024 × 1024`). `maxChildThreads` — If you set this option, the value must be an `integer` type representing the maximum number of threads each child decision tree can use. The default value is `1`. `maxDepth` — If you set this option, the value must be a positive integer. This value sets the maximum allowable depth of the decision tree. The default value is `3`. `maxThreads` — If you set this option, the option specifies the maximum number of parallel threads to use while the model trains decision trees. This value must be a positive integer. The default value is `16`. `metrics` — If you set this option to `true`, the model also calculates the percentage of samples that are correctly classified by the model for the random forest and saves this information in a system catalog table. The default value is `false`. `noSnapshot` — If you set this option to `true`, the data source must not change. In this case, the database does not create an intermediate table that stores the result of the specified SQL statement, which the model uses for training a random forest. Child decision trees always have this option set to `true`, so the database does not create a separate intermediate table for each decision tree. The default value is `false`. Setting this option to `true` is useful when the training set is fixed. If the training set is a table with modifications, set this option to `false` as the decision tree trainer uses different data sets in different parts of the tree. Likewise, if the training set consists of a query that returns 100 rows, then set this option to `false` because there is no guarantee that running that query twice generates the same 100 rows each time. `numSplits` — If you set this option, the value must be an `integer` type greater than 1. This value sets the maximum number of binary branches a continuous feature can consider. The default value is `32`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `requiredFeatures` — If you set this option, the option must be a comma-separated list of integers as strings representing specific features where the first feature has the value `1`. The model uses these features in every decision tree in the forest. The default behavior is that the decision tree in the forest can train on any feature in the list. `rowsPerChild` — If you set this option to a positive integer, the number represents the number of rows (from a random sample) to use for each decision tree. If you set this option to `0`, each child uses all available rows. The default value is `0`. You cannot set this option to a positive value if you also set the `fractionSelected` option. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `skipLimitCheck` — If you set this option to `true`, the model skips cardinality checks that throw errors when columns have too many values. The limit that this option checks is the same one that is specified by the `distinctCountLimit` option. This option defaults to `false`. `splitMetric` — If you set this option, the option controls which function the model uses to evaluate the quality of a split during tree construction. Supported options are: `gini_impurity` (measures impurity based on class distributions), `macro_f1` (uses macro-averaged F1 score to guide splits), `micro_f1` (uses micro-averaged F1 score to guide splits), and `weighted_f1` (uses class-frequency-weighted F1 score to guide splits). The default value is `gini_impurity`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. `weighted` — If you set this option, the model considers weights for labels. If you set this option value to `true`, you must specify an additional column as a `double` in the training data for label weights. Rows with the same labels must have the same weights. If you set this value to `auto`, the model calculates weights automatically by weighting each label according to the ratio of the count of the most frequent label to the count of the specified label. As a result, the most frequent label has a weight of `1.0`, and the other label weights are higher. This option defaults to `false`, which means all labels have equal weight. ## Regression Tree ### Model Options #### Optional `continuousFeatures` — If you set this option, the value must be a comma-separated list of the feature indexes that are continuous numeric variables. Indexes start with 1. In the default state, the model considers no features as continuous. `distinctCountLimit` — If you set this option, the value must be a positive integer. This value sets the limit for the number of distinct values a non-continuous feature and the label can contain. This option defaults to `256`. `enableResplits` — If you set this option, the value must be a `boolean` type that determines if the tree can reuse the same continuous feature multiple times along a single branch (e.g., split on `x1 < 7` and later `x1 < 3`). This action can capture more complex, range-specific relationships. The default value is `true`, meaning that continuous features remain available for additional splits after use, which allows the tree to create more complex decision boundaries. When you set this option to `false`, the model marks continuous features as exhausted after their first use, and the model cannot use them again in subsequent splits in the same tree. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `featureSubsetStrategy` — If you set this option, the option specifies how many features the regression tree should consider at each split from the still-available features. When this value is higher, the model has a higher accuracy and lower variance, but takes longer to train. You can specify this option either as an integer (e.g., `4`, meaning consider up to four features at each split) or one of these values: `all` (check every feature), `sqrt` (check up to the square root of the number of total features), and `one-third` (check up to one-third of the number of total features). The default value is `all`. `maxCellsToFetch` — If you set this option, the value must be a positive integer. Controls the chunking behavior when fetching feature values during model training. The limit represents the maximum number of data cells (calculated as the number of columns × number of rows) that the system can fetch in a single operation, not a byte limit. When the expected data size exceeds this threshold, the algorithm switches to database-based processing using SQL queries instead of in-memory processing. The default value is `33,554,432` cells (calculated as `32 × 1024 × 1024`). `maxDepth` — If you set this option, the value must be a positive integer. This value sets the maximum allowable depth of the decision tree (the maximum number of features to split on). The default is unspecified, which means there is no maximum depth. `maxRows` — If you set this option, the value must be a positive integer. This option limits the number of rows used for model training by creating a snapshot table with only the specified number of rows from the input query. This option cannot be used with `noSnapshot -> true` (attempting to set both results in an `invalid argument` error during model creation). When this option is unspecified, the model trains using all rows from the input query. `maxThreads` — If you set this option, the value must be a positive integer. This value indicates the maximum number of parallel threads to use while the model trains. The default value is `2`. `metrics` — If you set this option to `true`, the model also calculates the percentage of samples correctly classified by the model and saves this information in a system catalog table. The default value is `false`. `noSnapshot` — If you set this option to `true`, the database does not create an intermediate table that stores the result of the specified SQL statement, which the model uses for training. This option defaults to `false`. In this case, the database creates and uses the intermediate table. Setting this option to `true` is useful when the training set is fixed. If the training set is a table with modifications, set this option to `false`, as the decision tree trainer uses different data sets in different parts of the tree. Likewise, if the training set consists of a query that returns 100 rows, then set this option to `false` because there is no guarantee that executing that query twice generates the same 100 rows each time. `numSplits` — If you set this option, the value must be an integer greater than 1. This value sets the maximum number of binary branches a continuous feature can consider. The default value is `32`. `queryInternalParallelism` — If you set this option, the database appends the `USING PARALLELISM = ` clause to all intermediate SQL queries the model executes during training, where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `resplitDepth` — If you set this option, the value must be an `integer` type that sets the maximum depth at which tree nodes can be re-split during optimization. Controls how deep the algorithm searches for better split points. The default value is `6`. `resplitThreshold` — If you set this option, the value must be a `decimal` type that sets the minimum improvement threshold required to trigger a re-split operation. Lower values (e.g., `0.01`) allow more aggressive re-splitting but can increase training time. Higher values (e.g., `1.0`) require larger improvements to trigger re-splits. The default value is `0.1`. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `skipLimitCheck` — If you set this option to `true`, the model skips cardinality checks that throw errors when columns have too many values. The limit that this option checks is the same one that you specify using the `distinctCountLimit` option. The default value is `false`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Simple Linear Regression ### Model Options #### Optional `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `metrics` — If you set this option to `true`, the model collects quality metrics such as the coefficient of determination (R-squared) and the root mean squared error (RMSE). The default value is `false`. `normalize` — If you set this option to `true`, the model uses auto-scaling to compute the mean and standard deviation of each input feature to normalize data during training, making training more numerically stable. The model then unscales parameters so the persisted model operates in the original units. The default value is `true`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `threshold` — This option enables soft thresholding. If you specify this option, then the option must be a positive numeric value. After the model calculates the coefficients, if any are greater than the threshold value, the threshold value is subtracted from them. If any coefficients are less than the negation of the threshold value, the model adds the threshold value to them. For any coefficients that are between the negative and positive threshold values, the model sets those coefficients to zero. `yIntercept` — If you set this option, then the option must be a numeric value. The system forces the specific y-intercept (i.e., the model value when `x` is zero). ## Stacking ### Model Options #### Required `levelOneModel` — This option specifies the level-1 child models of the stacking model. You must specify this value as a JSON array, where each object in the array has the four fields `type` (required), `name`, `options`, `ignoreColumn`, and `extraCallArguments`. `levelZeroModels` — This option specifies the level-0 child models of the stacking model. You must specify this value as a JSON array, where each object in the array has the four fields `type` (required), `name`, `options`, `ignoreColumn`, and `extraCallArguments`. #### Optional `extraColumnCount` — If you set this option, the value must be an `integer` type that specifies how many non-feature columns there are in the input data. The default value is `0`. `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `hasLabelColumn` — If you set this option, the value must be a `boolean` type that specifies whether the input data includes a label column. The default value is `true`. `maxThreads` — If you set this option, the option specifies the maximum number of parallel threads to use while the model trains. This value must be a positive integer. The default value is `16`. `noSnapshot` — If you set this option to `true`, the data source must not change. In this case, the database does not create an intermediate table that stores the result of the specified SQL statement, which the model uses for training a random forest. Child decision trees always have this option set to `true`, so the database does not create a separate intermediate table for each decision tree. The default value is `false`. Setting this option to `true` can speed up training when the training set is fixed. `preservedColumnsForLevelOne` — If you set this option, this option specifies the columns from the original training data to pass as an input column to the level-1 model, in addition to the level-0 outputs. This value should be a comma-separated list of integers starting at 1. The default behavior is to preserve none of the columns. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Support Vector Machine ### Model Options #### Optional `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `functionN` — By default, SVM uses a linear kernel. If you use a different kernel, you must provide a list of functions that are summed together, just like with linear combination regression. You must specify the first function using a key named 'function1'. Subsequent functions must use keys with names that use subsequent values of N. You must specify functions in SQL syntax and use the variables x1, x2, … , xn to refer to the 1st, 2nd, and nth independent variables, respectively. You can specify the default linear kernel as: 'function1' → 'x1', 'function2' → 'x2', and so on. The model always adds a constant term equivalent to 'functionN' → '1.0' that you do not need to specify explicitly. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `metrics` — If you set this option to `true`, the model also calculates the percentage of samples that are correctly classified by the model and saves this information in a catalog table. This option defaults to `false`. `normalize` — If you set this option to `true`, the model automatically computes the mean and standard deviation of each feature and uses them to normalize the data during training. Defaults to `true`. `numEpochs` — If you set this option, the value must be a positive `integer` type representing the maximum number of IRLS iterations during training. The default value is `20`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `regularizationCoefficient` — If you set this option, the value must be a valid floating-point number. Use this option to control the balance of finding a wide margin and minimizing incorrectly classified points in the loss function. A larger (and positive) value makes having a wide margin around the hypersurface more important relative to the incorrectly classified points. Because of how the system implements SVM, the values for this option are likely different than values used in other common SVM implementations. The default value is `1.0 / 1000000.0`. `skipDropTable` — If you set this option to `false`, the database deletes any intermediate tables created during model training. If you set this option to `true`, the database prevents the deletion of any intermediate tables created during model training. The default value is `false`. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. ## Vector Autoregression ### Model Options #### Required `numLags` — Specify this option as a positive integer for the number of lags in the model. `numVariables` — Specify this option as a positive integer for the number of variables in the model. #### Optional `featureArray` — If you set this option to `true`, the model expects only one array-type input column instead of multiple columns of training data. Each array row in the input column must be the same size. The default value is `false`. `featureArrayElements` — If you set this option, the `featureArray` option must be set to `true`. The value must be a comma-separated list of integers representing indexes of the input array to use starting at index 1. The system uses all indexes of the input array by default. `loadBalance` — If you set this option, the database appends the `USING load_balance_shuffle = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified option value (`true` or `false`). The default value is unspecified. In this case, the database does not add this clause. `metrics` — If you set this option to `true`, the function collects the metric for the coefficient of determination (R-squared). The default value is `false`. `normalize` — If you set this option to `true`, the model uses auto-scaling to compute the mean and standard deviation of each input feature to normalize data during training, making training more numerically stable. The model then unscales parameters so the persisted model operates in the original units. The default value is `true`. `queryInternalParallelism` — If you set this option, the database appends the `USING parallelism = ` clause to all intermediate SQL queries the model executes during training where `value` is the specified positive integer value. The default value is unspecified. In this case, the database does not add this clause. `suppressArrayLengthCheck` — If you set this option, the `featureArray` option must be set to `true`. The system skips checking that the array length is the same size for all rows in the input. The default value is `false`. `threshold` — This option enables soft thresholding. If you specify this option, then the option must be a positive numeric value. After the model calculates the coefficients, if any of them are greater than the threshold value, the threshold value is subtracted from them. If any coefficients are less than the negation of the threshold value, the model adds the threshold value to them. For any coefficients that are between the negative and positive threshold values, the model sets those coefficients to zero. # Machine Learning Models Source: https://docs.ocient.com/machine-learning-models SQL reference for managing Ocient machine learning models, with syntax for CREATE, RENAME, EXPORT, RETRAIN, query, and DROP across 23 model types. functionality enables you to create a machine learning model, rename the model, export the syntax for the model creation, retrain the model, execute a query against the model, and drop the model. ## CREATE MLMODEL Train a new machine learning model of type `` on the result set returned by the SQL SELECT statement. After the database creates the model, `` becomes a callable function in SQL SELECT statements.  **Syntax** ```sql SQL theme={null} CREATE [ OR REPLACE ] MLMODEL TYPE ON( ) [options()] ``` #### model\_name | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | -------------------------------- | | `model name` | `VARCHAR` | The name of the model to create. | #### model\_type | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | --------------------------------------------- | | `model type` | `VARCHAR` | The type of machine learning model to create. | These models are supported. You can find full descriptions of each model in [Regression Models](/regression-models), [Classification Models](/classification-models), [Clustering and Dimension Reduction Models](/clustering-and-dimension-reduction-models), [Ensemble Models](/ensemble-models), or [Other Models](/other-models). * `SIMPLE LINEAR REGRESSION` * `MULTIPLE LINEAR REGRESSION` * `POLYNOMIAL REGRESSION` * `LINEAR COMBINATION REGRESSION` * `VECTOR AUTOREGRESSION` * `KMEANS` * `KNN (K Nearest Neighbors)` * `LOGISTIC REGRESSION` * `NAIVE BAYES` * `NONLINEAR REGRESSION` * `FEEDFORWARD NETWORK` * `PRINCIPAL COMPONENT ANALYSIS` * `LINEAR DISCRIMINANT ANALYSIS` * `SUPPORT VECTOR MACHINE` * `DECISION TREE` * `GAUSSIAN MIXTURE MODEL` * `ASSOCIATION RULES` * `GRADIENT BOOSTED TREES` * `REGRESSION TREE` * `BAGGING` * `BOOSTING` * `STACKING` #### option\_list | **Parameter** | **Data** **Type** | **Description** | | ------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `option_list` | `VARCHAR` | The options for the specified machine learning model that is specified as a comma-separated list in the format: `