LAT User-Defined Transformations

data pipelines are now the preferred method for loading data into the ocient system for details, see load data docid\ xq0tg7yph vn62uwufibu in addition to built in transformations, users can extend lat by writing custom user defined transformations (udts) to bootstrap udt development, contact {{ocient}} support udt interface a udt is implemented as a jmespath java function https //github com/burtcorp/jmespath java/blob/master/jmespath core/src/main/java/io/burt/jmespath/function/function java a function is comprised of three elements name the function’s name determines how it can be called in a user pipeline e g , negate arguments what inputs the function expects to be called with body the function’s implementation, including return value udt names must be globally unique lat validates udt names on startup to ensure that no two udt names collide with each other and that no udt name collides with an lat builtin transformation this example function demonstrates these concepts // negate(number) takes an input number and changes its sign // function name derived from the class name, negatefunction > negate public class negatefunction extends basefunction { public negatefunction() { super( // function arguments; this function takes exactly one argument of type number argumentconstraints typeof(jmespathtype number)); } // function body protected \<t> t callfunction(adapter\<t> runtime, list\<functionargument\<t>> arguments) { double original = runtime tonumber(arguments get(0) value()) doublevalue(); return runtime createnumber(original 1); } } udt packaging one or more udts can be packaged in a jar file for consumption by lat in addition to the udt implementations themselves, the jar file must contain a services file containing a mapping of all the udts the jar should expose for use by lat this service file must be located at meta inf/services/io burt jmespath function function in the resulting jar, and its contents should be as follows com ocient myudt negatefunction the file at the path meta inf/services/io burt jmespath function function should include the fully qualified name of one or more udts contained within this jar to be used by lat each udt should be specified on its own line in the file dependencies if the udt has dependencies on third party libraries, those libraries must be packaged in the same jar file this is commonly referred to as a "fat jar", "uber jar", or "jar with dependencies " instructions for building such a jar file with maven™ can be found here https //maven apache org/plugins/maven assembly plugin/single mojo html udt deployment lat can be configured to search for udt jars in a particular directory see lat source configuration docid\ z8gjws65x2ybq02bns gm for details lat can read multiple udts across multiple jar files on startup, lat will read all available udts into its configuration to update the set of udts available to lat, such as to add an additional udt jar, a process restart is required related links ingest data with legacy lat reference docid\ tt6tfoulap0mt aycm2ka