Loading and Transformation Ref...
Data Transformation with JMESP...

User-Defined Transformations

In addition to built-in transformations, users can extend LAT by writing custom user-defined transformations (UDTs).

To bootstrap UDT development, contact Support.

UDT Interface

A UDT is implemented as a JMESPath Java Function. A Function is comprised of three elements:

  1. name - the function’s name determines how it can be called in a user pipeline e.g., negate
  2. arguments - what inputs the function expects to be called with
  3. body - the function’s implementation, including return value

UDT names must be globally unique. LAT validates UDT names on startup to ensure that no two UDT names collide with each other and that no UDT name collides with an LAT builtin transformation.

This example function demonstrates these concepts:

Java


UDT Packaging

One or more UDTs can be packaged in a jar file for consumption by LAT. In addition to the UDT implementations themselves, the jar file must contain a services file containing a mapping of all the UDTs the jar should expose for use by LAT. This service file must be located at META-INF/services/io.burt.jmespath.function.Function in the resulting jar, and its contents should be as follows:

Text


The file at the path META-INF/services/io.burt.jmespath.function.Function should include the fully qualified name of one or more UDTs contained within this jar to be used by LAT. Each UDT should be specified on its own line in the file.

Dependencies

If the UDT has dependencies on third party libraries, those libraries must be packaged in the same jar file. This is commonly referred to as a "fat jar", "uber jar", or "jar with dependencies." Instructions for building such a jar file with Maven™ can be found here.

UDT Deployment

LAT can be configured to search for UDT jars in a particular directory. See Source Configuration for details. LAT can read multiple UDTs across multiple jar files.

On startup, LAT will read all available UDTs into its configuration. To update the set of UDTs available to LAT, such as to add an additional UDT jar, a process restart is required.

Related Links

Load Data