LAT Transform Configuration

data pipelines are now the preferred method for loading data into the ocient system for details, see load data docid 1nw163anqyhk58cqmepjs transform a transform configuration object required keys transform configuration source only lat transform configuration docid 4 64z9kaveebxvnxk36vb file sources only lat transform configuration docid 4 64z9kaveebxvnxk36vb transform topics a collection of topics and their associated configuration keys are topic names and values are lat transform configuration docid 4 64z9kaveebxvnxk36vb each \<topic> set as a key in this object represents an {{kafka}} topic defined in kafka when loading from an s3 or local file source type, transform file groups should be used instead of transform topics each key in the file group must match a file group name defined in the source section of the pipeline configuration type object required yes default transform database a database to be used to fully qualify any table names that are not fully qualified for example, if transform database is set to mydatabase , a table name of the form schema table will become mydatabase schema table type string required no default null transform schema a schema to be used to fully qualify any table names that are not fully qualified specifying this property requires that lat transform configuration docid 4 64z9kaveebxvnxk36vb is specified for example, if transform database is set to mydatabase and transform schema is set to myschema a table name of the form table will become mydatabase myschema table a table name of the form schema table will become mydatabase schema table a table name of the form database schema table will stay as database schema table type string required no default null kafka load transform example { "transform" { "topics" { "topic 1" { "tables" { } }, "topic 2" { "tables" { } } } } } file based load transform example unlike kafka, file loads define file groups in the source section of the pipeline configuration the "file groups" defined in the source and transform sections must match { "source" { "type" "s3", "file groups" { "file group 1" { }, "file group 2" { }, } } "transform" { "file groups" { "file group 1" { "tables" { } }, "file group 2" { "tables" { } } } } } topics topic configuration objects required keys lat transform configuration docid 4 64z9kaveebxvnxk36vb for file based loads, topics are replaced by file groups , but all other settings are equivalent transform topics \<topic> filter a record filter to apply at the topic level see lat record filtering docid\ nk5elq kt18y92pns06nw for details type string required yes default transform topics \<topic> tables a collection of tables and their associated configuration keys are table names and values are lat transform configuration docid 4 64z9kaveebxvnxk36vb type object required yes default tables table configuration objects required keys lat transform configuration docid 4 64z9kaveebxvnxk36vb transform topics \<topic> tables \<table> filter a record filter to apply at the table level see lat record filtering docid\ nk5elq kt18y92pns06nw for details type string required yes default transform topics \<topic> tables \<table> columns a collection of columns and their associated configurations keys are table names and values are lat transform configuration docid 4 64z9kaveebxvnxk36vb columns column transformation configurations required keys lat transform configuration docid 4 64z9kaveebxvnxk36vb transform topics \<topic> tables \<table> columns \<column> a column transformation keyed by a column name a column’s value is defined as a transformation expression the expression will query the record and return a value that is loaded into the associated column the grammar of these expressions uses jmespath https //jmespath org/ enhanced with some custom {{ocient}} transformations and user defined transformations (udts) type string required yes default complex transform example { "transform" { "database" "adtechdb", "schema" "adtech", "topics" { "dsp auctions" { "tables" { "auction" { "columns" { "auctionid" "auctionid", "created" "created", "bidder nodeid" "bidder nodeid", "deals dealid" "deals\[] dealid" } }, "trafficsource" { "columns" { "auctionid" "auctionid", "created" "created", "sourcesellerid" "trafficsource sourcesellerid", "domainname" "trafficsource domainname", "contextualdataset externalentryid" "trafficsource sitecontextualprofile contextualdataset\[] entries\[] externalentryid\[]", "dimensions hw" "join('x', \[dimensions height, dimensions width])" } }, "feedback" { "columns" { "auctionid" "auctionid", "created" "created", "received" "received", "campaignid" "explode(auctionresponsefeedbacks\[] campaignid)", "won" "explode(auctionresponsefeedbacks\[] won)", "winningbid" "explode(auctionresponsefeedbacks\[] winningbid)", "sourcelossreasoncode" "explode(auctionresponsefeedbacks\[] sourcelossreasoncode)" } } } } } } } related links ingest data with legacy lat reference docid ul5nilgyd7y wjccg59d