LAT Transform Configuration

data pipelines are now the preferred method for loading data into the ocient system for details, see docid\ zncvnrhsf6fg1yvqk6mxt transform a transform configuration object required keys transform configuration source only docid\ xk acu9i3s5tqrqgrz26v file sources only docid\ xk acu9i3s5tqrqgrz26v transform topics a collection of topics and their associated configuration keys are topic names and values are docid\ xk acu9i3s5tqrqgrz26v each \<topic> set as a key in this object represents an {{kafka}} topic defined in kafka when loading from an s3 or local file source type, transform file groups should be used instead of transform topics each key in the file group must match a file group name defined in the source section of the pipeline configuration left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type transform database a database to be used to fully qualify any table names that are not fully qualified for example, if transform database is set to mydatabase , a table name of the form schema table will become mydatabase schema table 0 left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type transform schema a schema to be used to fully qualify any table names that are not fully qualified specifying this property requires that docid\ xk acu9i3s5tqrqgrz26v is specified for example, if transform database is set to mydatabase and transform schema is set to myschema a table name of the form table will become mydatabase myschema table a table name of the form schema table will become mydatabase schema table a table name of the form database schema table will stay as database schema table left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type kafka load transform example { "transform" { "topics" { "topic 1" { "tables" { } }, "topic 2" { "tables" { } } } } } file based load transform example unlike kafka, file loads define file groups in the source section of the pipeline configuration the "file groups" defined in the source and transform sections must match { "source" { "type" "s3", "file groups" { "file group 1" { }, "file group 2" { }, } } "transform" { "file groups" { "file group 1" { "tables" { } }, "file group 2" { "tables" { } } } } } topics topic configuration objects required keys docid\ xk acu9i3s5tqrqgrz26v for file based loads, topics are replaced by file groups , but all other settings are equivalent transform topics \<topic> filter a record filter to apply at the topic level see docid\ wx0dpn9ymwym f vv4ea for details 372,372 left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type transform topics \<topic> tables a collection of tables and their associated configuration keys are table names and values are docid\ xk acu9i3s5tqrqgrz26v left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type tables table configuration objects required keys docid\ xk acu9i3s5tqrqgrz26v transform topics \<topic> tables \<table> filter a record filter to apply at the table level see docid\ wx0dpn9ymwym f vv4ea for details left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type transform topics \<topic> tables \<table> columns a collection of columns and their associated configurations keys are table names and values are docid\ xk acu9i3s5tqrqgrz26v columns column transformation configurations required keys docid\ xk acu9i3s5tqrqgrz26v transform topics \<topic> tables \<table> columns \<column> a column transformation keyed by a column name a column’s value is defined as a transformation expression the expression will query the record and return a value that is loaded into the associated column the grammar of these expressions uses https //jmespath org/ enhanced with some custom {{ocient}} transformations and user defined transformations (udts) left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type left unhandled content type complex transform example { "transform" { "database" "adtechdb", "schema" "adtech", "topics" { "dsp auctions" { "tables" { "auction" { "columns" { "auctionid" "auctionid", "created" "created", "bidder nodeid" "bidder nodeid", "deals dealid" "deals\[] dealid" } }, "trafficsource" { "columns" { "auctionid" "auctionid", "created" "created", "sourcesellerid" "trafficsource sourcesellerid", "domainname" "trafficsource domainname", "contextualdataset externalentryid" "trafficsource sitecontextualprofile contextualdataset\[] entries\[] externalentryid\[]", "dimensions hw" "join('x', \[dimensions height, dimensions width])" } }, "feedback" { "columns" { "auctionid" "auctionid", "created" "created", "received" "received", "campaignid" "explode(auctionresponsefeedbacks\[] campaignid)", "won" "explode(auctionresponsefeedbacks\[] won)", "winningbid" "explode(auctionresponsefeedbacks\[] winningbid)", "sourcelossreasoncode" "explode(auctionresponsefeedbacks\[] sourcelossreasoncode)" } } } } } } } related links docid\ zjlpeecj74 lp0ciflqh0