LAT Transform Configuration

data pipelines are now the preferred method for loading data into the ocient system for details, see docid\ xq0tg7yph vn62uwufibu transform a transform configuration object required keys transform configuration source only /#transformtopics file sources only /#transformtopics transform topics a collection of topics and their associated configuration keys are topic names and values are /#topics each \<topic> set as a key in this object represents an {{kafka}} topic defined in kafka when loading from an s3 or local file source type, transform file groups should be used instead of transform topics each key in the file group must match a file group name defined in the source section of the pipeline configuration type object required yes default transform database a database to be used to fully qualify any table names that are not fully qualified for example, if transform database is set to mydatabase , a table name of the form schema table will become mydatabase schema table type string required no default null transform schema a schema to be used to fully qualify any table names that are not fully qualified specifying this property requires that /#transformdatabase is specified for example, if transform database is set to mydatabase and transform schema is set to myschema a table name of the form table will become mydatabase myschema table a table name of the form schema table will become mydatabase schema table a table name of the form database schema table will stay as database schema table type string required no default null kafka load transform example { "transform" { "topics" { "topic 1" { "tables" { } }, "topic 2" { "tables" { } } } } } file based load transform example unlike kafka, file loads define file groups in the source section of the pipeline configuration the "file groups" defined in the source and transform sections must match { "source" { "type" "s3", "file groups" { "file group 1" { }, "file group 2" { }, } } "transform" { "file groups" { "file group 1" { "tables" { } }, "file group 2" { "tables" { } } } } } topics topic configuration objects required keys /#transformtopicstopictables for file based loads, topics are replaced by file groups , but all other settings are equivalent transform topics \<topic> filter a record filter to apply at the topic level see docid 07kl0zrg6cqqarqhmmmif for details type string required yes default transform topics \<topic> tables a collection of tables and their associated configuration keys are table names and values are /#columns type object required yes default tables table configuration objects required keys /#transformtopicstopictablestablecolumns transform topics \<topic> tables \<table> filter a record filter to apply at the table level see docid 07kl0zrg6cqqarqhmmmif for details type string required yes default transform topics \<topic> tables \<table> columns a collection of columns and their associated configurations keys are table names and values are /#columns columns column transformation configurations required keys /#transformtopicstopictablestablecolumnscolumn transform topics \<topic> tables \<table> columns \<column> a column transformation keyed by a column name a column’s value is defined as a transformation expression the expression will query the record and return a value that is loaded into the associated column the grammar of these expressions uses https //jmespath org/ enhanced with some custom {{ocient}} transformations and user defined transformations (udts) type string required yes default complex transform example { "transform" { "database" "adtechdb", "schema" "adtech", "topics" { "dsp auctions" { "tables" { "auction" { "columns" { "auctionid" "auctionid", "created" "created", "bidder nodeid" "bidder nodeid", "deals dealid" "deals\[] dealid" } }, "trafficsource" { "columns" { "auctionid" "auctionid", "created" "created", "sourcesellerid" "trafficsource sourcesellerid", "domainname" "trafficsource domainname", "contextualdataset externalentryid" "trafficsource sitecontextualprofile contextualdataset\[] entries\[] externalentryid\[]", "dimensions hw" "join('x', \[dimensions height, dimensions width])" } }, "feedback" { "columns" { "auctionid" "auctionid", "created" "created", "received" "received", "campaignid" "explode(auctionresponsefeedbacks\[] campaignid)", "won" "explode(auctionresponsefeedbacks\[] won)", "winningbid" "explode(auctionresponsefeedbacks\[] winningbid)", "sourcelossreasoncode" "explode(auctionresponsefeedbacks\[] sourcelossreasoncode)" } } } } } } } related links docid\ tt6tfoulap0mt aycm2ka