Spring Kafka - Avro Bijection Example

Twitter Bijection is an invertible function library that converts back and forth between two types. It supports a number of types including Apache Avro.

In the following tutorial, we will configure, build and run an example in which we will send/receive an Avro message to/from Apache Kafka using Bijection, Apache Avro, Spring Kafka, Spring Boot and Maven.

General Project Setup

Tools used:

Twitter Bijection 0.9

Apache Avro 1.8

Spring Kafka 1.2

Spring Boot 1.5

Maven 3.5

We base this example on a previous Spring Kafka Avro serializer/deserializer example in which we used the Avro API’s to serialize and deserialize objects. For this code sample, we will be using the Bijection APIs which are a bit easier to use as we will see further down below.

Starting point is again the user.avsc schema from the Avro getting started guide. It describes the fields and their types of a User type.

We setup our project using Maven. In the POM file we add the bijection-avro_2.11 dependency. The artifactId suffix of the dependency (in this case _2.11) highlights the Scala version used to compile the JAR.

Note that we choose the 2.11 version of bijection-avro since spring-kafka-test includes a dependency on the 2.11 version of the scala-library.

Generation of the Avro User class is done by executing below Maven command. The result is a User class that contains the schema and Builder methods.

mvn generate-sources

Producing Avro Messages to a Kafka Topic

Serializing an Avro message to a byte[] array using Bijection can be achieved in just two lines of code as shown below.

We first create an Injection which is an object that can make the conversion in one way or the other. This is done by calling the static toBinary() method on the GenericAvroCodecs class. The result is an Injection capable of serializing and deserializing a generic Avro record using org.apache.avro.io.BinaryEncoder. As an input parameter, we need to supply the Avro schema which we get from the passed object.

The ‘apply()’ method is then used to create the Byte array which is returned.

Test Sending and Receiving Avro Messages on Kafka

Using @Before we wait until all the partitions are assigned to our Receiver by looping over the available ConcurrentMessageListenerContainer (if we don’t do this the message will already be sent before the listeners are assigned to the topic).

In the testReceiver() test case an Avro User object is created using the Builder methods. This user is then sent to the 'avro-bijection.t' topic. Finally, the CountDownLatch from the Receiver is used to verify that a message was successfully received.

packagecom.codenotfound.kafka;importstaticorg.assertj.core.api.Assertions.assertThat;importjava.util.concurrent.TimeUnit;importorg.junit.Before;importorg.junit.ClassRule;importorg.junit.Test;importorg.junit.runner.RunWith;importorg.springframework.beans.factory.annotation.Autowired;importorg.springframework.boot.test.context.SpringBootTest;importorg.springframework.kafka.config.KafkaListenerEndpointRegistry;importorg.springframework.kafka.listener.MessageListenerContainer;importorg.springframework.kafka.test.rule.KafkaEmbedded;importorg.springframework.kafka.test.utils.ContainerTestUtils;importorg.springframework.test.context.junit4.SpringRunner;importcom.codenotfound.kafka.consumer.Receiver;importcom.codenotfound.kafka.producer.Sender;importexample.avro.User;@RunWith(SpringRunner.class)@SpringBootTestpublicclassSpringKafkaApplicationTest{@AutowiredprivateSendersender;@AutowiredprivateReceiverreceiver;@AutowiredprivateKafkaListenerEndpointRegistrykafkaListenerEndpointRegistry;@ClassRulepublicstaticKafkaEmbeddedembeddedKafka=newKafkaEmbedded(1,true,"avro-bijection.t");@BeforepublicvoidsetUp()throwsException{// wait until the partitions are assignedfor(MessageListenerContainermessageListenerContainer:kafkaListenerEndpointRegistry.getListenerContainers()){ContainerTestUtils.waitForAssignment(messageListenerContainer,embeddedKafka.getPartitionsPerTopic());}}@TestpublicvoidtestReceiver()throwsException{Useruser=User.newBuilder().setName("John Doe").setFavoriteColor("blue").setFavoriteNumber(null).build();sender.send(user);receiver.getLatch().await(10000,TimeUnit.MILLISECONDS);assertThat(receiver.getLatch().getCount()).isEqualTo(0);}}

Note that the sample code also contains AvroSerializerTest and AvroDeserializerTest unit test cases to verify the serialization classes.

Trigger the above test case using a command prompt and following Maven command:

mvn test

Maven will do the necessary and the outcome should be a successful build as shown below: