Kafka Connect JSLT Single Message Transform

willhaben

Last update: May 25, 2022

Related tags

Kotlin Projects wh-kafka-connect-jslt-transform

Overview

Kafka Connect JSLT Single Message Transform

This is an implementation of the Kafka Connect SMT interface to offer transformation capabilities using the Schibsted JSLT library.

Build

The library uses Gradle to build the JAR.

Install latest Java SDK.
Checkout the Git repository and change to its root folder.
Execute ./gradlew build

The JAR can then be found in the build/libs/ subfolder.

Install

After the JAR was build as described above, copy it to your Kafka Connect instance into one of the directories listed in the plugin.path property in the connect worker configuration file.

Make sure to do this on all Kafka Connect worker nodes!

See the Confluent Kafka Connect Plugins Userguide for more details.

Usage

Connector Configuration

The transformer expects the jslt attribute in the connector config JSON. In the following example the dot means it will not perform any transformation but this string can be any valid JSLT expression.

Note that Json does not support multiline strings. So linebreaks and quotes must be escaped with a backslash (e.g. " -> \")

{
  "name": ...,
  "config": {
    ...,
    "transforms": "jsltTransform",
    "transforms.jsltTransform.type": "at.willhaben.kafka.connect.transforms.jslt.JsltTransform$Value",
    "transforms.jsltTransform.jslt": "."
  }
}

Note that the transformer only supports some kind of structured input. So make sure that there is a converter class (e.g. AvroConverter, JsonConverter) like "value.converter": "io.confluent.connect.avro.AvroConverter" that provides the data in a structured format.

The following links are helpful to learn more about JSLT:

JSLT Tutorial and Examples: examples of the JSLT language features
JSLT Function Reference: Documentation of the included functions
JSLT Demo Playground: comes in handy to try out the logic

Example JSLT

The following JSLT takes a nested structure as input and

adds a new constant field producer_team
maps the values of the field customer_status from a numeric representation to a code, using custom logic
pseudonymizes the customer_id field
adds a list of objects field locations based on the separate invoice/delivery address fields of the input
extracts and flattens the two attributes customer_type and customer_class

Input Record

{
  "cusomer_id": 123456,
  "customer_status": 2,
  "invoice_address_street": "McDuck Manor",
  "invoice_address_zip_code": "1312",
  "invoice_address_city": "Duckburg",
  "delivery_address_street": "Webfoot Walk",
  "delivery_address_zip_code": "1313",
  "delivery_address_city": "Duckburg",
  "attributes": {
    "customer_type": "C2C",
    "customer_class": "A",
    "last_order": "2022-03-10T18:25:43.511Z"
  }
}

JSLT

{
  def map_status(status)
  if ($status == 0)
    "ACTIVE"
  else if ($status == 1)
    "INACTIVE"
  else
    "UNDEFINED"

  "customer_id": sha256-hex(.customer_id),
  "customer_status_code": map_status(.customer_status),
  "locations": [
    {
      "zip_code": .invoice_address_zip_code,
      "city": .invoice_address_city
    },
    {
      "zip_code": .delivery_address_zip_code,
      "city": .delivery_address_city
    }
  ],
  "customer_type": .attributes.customer_type,
  "customer_class": .attributes.customer_class,
  "producer_team": "us.california.burbank.disney"
}

Output Record

{
  "customer_id": "8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92",
  "customer_status_code": "INACTIVE",
  "locations": [
    {
      "zip_code": "1312",
      "city": "Duckburg"
    },
    {
      "zip_code": "1313",
      "city": "Duckburg"
    }
  ],
  "customer_type": "C2C",
  "customer_class": "A",
  "producer_team": "us.california.burbank.disney"
}

Additional Notes

The JSLT library works with the JsonNode class to perform the transformation.

This class uses quite generic data types and the conversion to/from JsonNode could lead to some changes in the returned datatype. This might especially an issue with floating point values!

You might also like...

A Kotlin/Java library to connect directly to an Android device without an adb binary or an ADB server

dadb Blog Post: Our First Open-Source Project A Kotlin/Java library to connect directly to an Android device without an adb binary or an ADB server de

791 Dec 20, 2022

A simple Kotlin class to use to connect to a MongoDB collection.

mongodb-kotlin A simple Kotlin class to use to connect to a MongoDB collection. Once you have created a MongoDB project, choose Build a Database: Next

2 Oct 23, 2022

Therapeutic is a platform to help easily connect patients or generally anyone struggling to get through tough times with motivating content and professional therapists.

Therapeutic Apk - https://github.com/develNerd/Therapeutic/blob/main/androidApp/release/androidApp-release6.apk Therapeutic is a Kotlin Mobile Multipl

1 Nov 23, 2022

This app is a single word scrambler app. The app displays one scramble word at a time, and the player has to guess the word using all the letters from the scrambled word. The player scores points if the word is correct, otherwise the player can play any number of times. There are 10 words per game.

Unscramble App Starter code for Android Basics codelab - Store the data in a ViewModel Unscramble is a single player game app that displays scrambled

5 Dec 6, 2021

Learn how to make an app designed for single-screen devices shine when running on foldable and dual-screen devices

dcberlin21-workshop Make your app shine om foldable devices with the samples we have here. Related links SDK open-source code SDK samples (Kotlin) App

3 Oct 26, 2021

A single screen app learn in google basic Android Development course.

Project: Lemonade App - Starter Code Starter code for the first independent project for Android Basics in Kotlin Introduction This is the starter code

0 Dec 19, 2022

In this single activity app. i was trying to practice on ViewModel and Livedata

CalwithViewModel In this single activity app. i was trying to practice on ViewModel and Livedata Min Api Level : 19 Setup Requirements Android device

1 Feb 16, 2022

Small Kafka Playground to play around with Test Containers, and KotlinX Coroutines bindings while reading Kafka Definite Guide V2

KafkaPlayground Small playground where I'm playing around with Kafka in Kotlin and the Kafka SDK whilst reading the Kafka book Definite Guide from Con

34 Dec 30, 2022

Kafka bindings for Kotlin `suspend`, and Kafka streaming operators for KotlinX Flow.

Module kotlin-kafka Rationale Goals Example This project is still under development, andd started as a playground where I was playing around with Kafk

34 Dec 30, 2022

kafka test with embedded kafka

kafka-test Requirements running Kafka on localhost:9092 How to use cat sampleuser.json | http POST localhost:9000/produce or use runConfiguration ./.r

0 Dec 9, 2021

Kafka-hot-and-cold-retries - Demo project for elaborating how hot and cold retries can be applied in Kafka

Apache Kafka® - Hot and Cold Retries A demo project for elaborating how hot and

11 Jun 10, 2022

Messaging API: Connect to PostgreSQL and Kafka to obtain and manage cars inventory

Messaging-API In this simple API, we connect to PostgreSQL and Kafka to obtain a

0 Feb 18, 2022

Kafka Connect connector for CØSMOS based blockchains.

CØSMOS Kafka Connector Kafka Connect connector for CØSMOS based blockchains. Purpose The Kafka Connect CØSMOS Source connector is a Kafka connector us

6 Nov 23, 2022

A tiny Android app that can receive WhatsApp group message export files and parse message history stats.

2 May 3, 2022

This is a easy way to publish MQTT message and receive MQTT message

SMQ-CLIENT This is a easy way to publish MQTT message and receive MQTT message This is provider a spring stater for quick use Recive message form the

1 Apr 25, 2022

Connect is an Android Application to connect people for a project and can then work on the application to efficiently complete the project

1 Jan 9, 2022

A library to take picture easy, transform your data in different format and save photos in your device

Comments

Option to Mark Fields as Mandatory

Firstly a quick thanks. After struggling with the SMTs currently on offer.. JSLT for Kafka Connectors is the silver bullet addressing most of our needs. All except for this one remaining issue; all fields produced by the transform are set as optional in the Schema Registry (we're using AVRO).

Ideally the optional state of fields would be configurable and would work something like Confluent's InsertField SMT which lets you mark fields as mandatory by suffixing the field name with a ! - see InsertField.

opened by albp-rmt 1

Kafka Connect JSLT Single Message Transform

Related tags

Overview

Kafka Connect JSLT Single Message Transform

Build

Install

Usage

Connector Configuration

Example JSLT

Additional Notes

You might also like...

A Kotlin/Java library to connect directly to an Android device without an adb binary or an ADB server

A simple Kotlin class to use to connect to a MongoDB collection.

Therapeutic is a platform to help easily connect patients or generally anyone struggling to get through tough times with motivating content and professional therapists.

This app is a single word scrambler app. The app displays one scramble word at a time, and the player has to guess the word using all the letters from the scrambled word. The player scores points if the word is correct, otherwise the player can play any number of times. There are 10 words per game.

Learn how to make an app designed for single-screen devices shine when running on foldable and dual-screen devices

A single screen app learn in google basic Android Development course.

In this single activity app. i was trying to practice on ViewModel and Livedata

Small Kafka Playground to play around with Test Containers, and KotlinX Coroutines bindings while reading Kafka Definite Guide V2

Kafka bindings for Kotlin `suspend`, and Kafka streaming operators for KotlinX Flow.

kafka test with embedded kafka

Kafka-hot-and-cold-retries - Demo project for elaborating how hot and cold retries can be applied in Kafka

Messaging API: Connect to PostgreSQL and Kafka to obtain and manage cars inventory

Kafka Connect connector for CØSMOS based blockchains.

A tiny Android app that can receive WhatsApp group message export files and parse message history stats.

This is a easy way to publish MQTT message and receive MQTT message

Connect is an Android Application to connect people for a project and can then work on the application to efficiently complete the project

A library to take picture easy, transform your data in different format and save photos in your device

🌠 Transform into a different view or activity using morphing animations.

Transform java callback to kotlin suspend method.

Comments

Option to Mark Fields as Mandatory

Releases(1.1.0)

1.1.0(Jun 1, 2022)

1.0.1(May 22, 2022)

1.0.0(Mar 18, 2022)

Owner

willhaben

kafka test with embedded kafka

Kafka-hot-and-cold-retries - Demo project for elaborating how hot and cold retries can be applied in Kafka

Kafka Connect connector for CØSMOS based blockchains.

Counterpart of onTouchEvent for Jetpack Compose and transform gesture with specific number of pointers

Kafka Streams Processor to unwrap CØSMOS blocks into CØSMOS transactions

Send Whatsapp Message Without Saving Mobile Number

A backend service that generates a random 32-chars length message and its sha256 hashcode and put them to DB

This application is used to adjust the scaling of kubernetes deployments depending on the message-count of given RabbitMQ queues.

Customtoast is a android library for show your toast message in a custom UI.

The behavior of Couchbase Lite C when attempting to connect to a Sync Gateway