LAT Record Filtering

data pipelines are now the preferred method for loading data into the ocient system for details, see docid\ xq0tg7yph vn62uwufibu incoming records can optionally be filtered by a jmespath expression added with an additional filter key in a pipeline configuration { "transform" { "topics" { "my topic" { "filter" "\<jmespath expression>", "tables" { "my table" { "filter" "\<jmespath expression>", "columns" {} } } } } } } records that pass through the filter (the filter expression evaluates to true) are processed as normal, while records that fail the filter (the filter expression evaluates to false) are dropped note that filters can be applied at the topic or table levels a record needs to pass through all configured filters in order to be loaded to a table the filter jmespath expression must evaluate to a boolean value examples of this are jmespath comparators or functions that return a boolean like contains , starts with , and ends with example pipeline { "transform" { "topics" { "topic0" { "filter" "integer0 == `2`", "tables" { "table0" { "filter" "integer1 == `3`", "columns" { "col0" "integer0", "col1" "integer1" } } } } } } } input \[ { "integer0" 0, "integer1" 3 }, { "integer0" 2, "integer1" 1 }, { "integer0" 2, "integer1" 3 } ] output \[ { "integer0" 2, "integer1" 3 } ] related links docid\ tt6tfoulap0mt aycm2ka