> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ocient.com/llms.txt
> Use this file to discover all available pages before exploring further.

# LAT Transform Configuration

export const Ocient = "Ocient®";

export const Kafka = "Apache® Kafka®";

<Warning>
  Data Pipelines are now the preferred method for loading data into the {Ocient} System. For details, see [Load Data](/load-data).
</Warning>

## `transform`

A transform configuration object.

Required keys:

* Transform Configuration source only: [transform.topics](#transform-topics)
* File sources only: [transform.file\_groups](#transform-topics)

### `transform.topics` \[#transform-topics]

A collection of topics and their associated configuration. Keys are topic names and values are [topic configuration objects](#topics). Each `<topic>` set as a key in this object represents an {Kafka} topic defined in Kafka.

<Info>
  When loading from an `s3` or `local` File Source type, `transform.file_groups` should be used instead of `transform.topics`. Each key in the `file_group` must match a `file_group_name` defined in the Source section of the pipeline configuration.
</Info>

| **Type:**     | object |
| ------------- | ------ |
| **Required:** | Yes    |
| **Default:**  |        |

### `transform.database` \[#transform-database]

A database to be used to fully qualify any table names that are not fully qualified.

For example, if `transform.database` is set to `myDatabase`, a table name of the form `schema.table` will become `myDatabase.schema.table`.

| **Type:**     | string |
| ------------- | ------ |
| **Required:** | No     |
| **Default:**  | null   |

### `transform.schema`

A schema to be used to fully qualify any table names that are not fully qualified. Specifying this property requires that [transform.database](#transform-database)  is specified.

For example, if `transform.database` is set to `myDatabase` and `transform.schema` is set to `mySchema`:

* A table name of the form `table` will become `myDatabase.mySchema.table`
* A table name of the form `schema.table` will become `myDatabase.schema.table`
* A table name of the form `database.schema.table` will stay as `database.schema.table`

| **Type:**     | string |
| ------------- | ------ |
| **Required:** | No     |
| **Default:**  | null   |

#### Kafka Load Transform Example

```json JSON theme={null}
{
    ...
    "transform": {
        "topics": {
            "topic_1": {
                "tables": {
                    ...
                }
            },
            "topic_2": {
                "tables": {
                    ...
                }
            }
        }
    }
}
```

#### File Based Load Transform Example

Unlike Kafka, File loads define file groups in the source section of the pipeline configuration. The "file\_groups" defined in the source and transform sections must match.

```json JSON theme={null}
{
    "source": {
        "type": "s3",
        ...
        "file_groups": {
            "file_group_1": {
                ...
            },
            "file_group_2": {
                ...
            },
        }
    }
    "transform": {
        "file_groups": {
            "file_group_1": {
                "tables": {
                    ...
                }
            },
            "file_group_2": {
                "tables": {
                    ...
                }
            }
        }
    }
}
```

### Topics

Topic configuration objects.

Required keys:

* [transform.topics.\<topic>.tables ](#transform-topicstopictables)

<Info>
  For file based loads, `topics` are replaced by `file_groups`, but all other settings are equivalent.
</Info>

#### `transform.topics.<topic>.filter`

A record filter to apply at the topic level. See [LAT Record Filtering](/lat-record-filtering) for details.

| **Type:**     | string |
| ------------- | ------ |
| **Required:** | Yes    |
| **Default:**  |        |

#### `transform.topics.<topic>.tables`

A collection of tables and their associated configuration. Keys are table names and values are [columns configuration](#columns).

| **Type**:     | object |
| ------------- | ------ |
| **Required**: | Yes    |
| **Default**:  |        |

### Tables

Table configuration objects.

Required keys:

* [transform.topics.\<topic>.tables.\<table>.columns](#transform-topicstopictablestablecolumns)

#### `transform.topics.<topic>.tables.<table>.filter`

A record filter to apply at the table level. See [LAT Record Filtering](/lat-record-filtering) for details.

| **Type**:     | string |
| ------------- | ------ |
| **Required**: | Yes    |
| **Default**:  |        |

#### `transform.topics.<topic>.tables.<table>.columns`

A collection of columns and their associated configurations. Keys are table names and values are [Columns](#columns).

### Columns

Column transformation configurations.

Required keys:

* [transform.topics.\<topic>.tables.\<table>.columns.\<column> ](#transform-topicstopictablestablecolumnscolumn)

#### `transform.topics.<topic>.tables.<table>.columns.<column>`

A column transformation keyed by a column name. A column’s value is defined as a transformation expression. The expression will query the record and return a value that is loaded into the associated column. The grammar of these expressions uses [JMESPath](https://jmespath.org/) enhanced with some custom Ocient transformations and User Defined Transformations (UDTs).

| **Type**:     | string |
| ------------- | ------ |
| **Required**: | Yes    |
| **Default**:  |        |

## Complex Transform Example

```json JSON theme={null}
{
    "transform": {
        "database": "adtechdb",
        "schema": "adtech",
        "topics" : {
            "dsp.Auctions" : {
                "tables" : {
                    "auction" : {
                        "columns" : {
                            "auctionid" : "auctionId",
                            "created" : "created",
                            "bidder_nodeid" : "bidder.nodeId",
                            "deals_dealid" : "deals[].dealId"
                        }
                    },
                    "trafficSource" : {
                        "columns" : {
                            "auctionid" : "auctionId",
                            "created" : "created",
                            "sourcesellerid" : "trafficSource.sourceSellerId",
                            "domainname" : "trafficSource.domainName",
                            "contextualdataset_externalentryid" : "trafficSource.siteContextualProfile.contextualDataSet[].entries[].externalEntryId[]",
                            "dimensions_hw" : "join('x', [dimensions.height, dimensions.width])"
                        }
                    },
                    "feedback" : {
                        "columns" : {
                            "auctionid" : "auctionId",
                            "created" : "created",
                            "received" : "received",
                            "campaignid" : "EXPLODE(auctionResponseFeedbacks[].campaignId)",
                            "won" : "EXPLODE(auctionResponseFeedbacks[].won)",
                            "winningbid" : "EXPLODE(auctionResponseFeedbacks[].winningBid)",
                            "sourcelossreasoncode" : "EXPLODE(auctionResponseFeedbacks[].sourceLossReasonCode)"
                        }
                    }
                }
            }
        }
    }
}
```

### Related Links

[Ingest Data with Legacy LAT Reference](/ingest-data-with-legacy-lat-reference)