Once you have the Confluent Platform up and running, either
locally for testing or in a
production setting, you’re ready to start using Kafka to
produce and consume data. This document helps you understand the tools available
for interacting with Kafka and choose the best one for your application. Before
getting started, make sure to understand the components of the
Confluent Platform, especially the
Schema Registry.

There are multiple ways for applications to interact with Kafka through the
Confluent Platform. Whichever method you choose for your application, the most
important factor is to ensure that your application is coordinating with the
Schema Registry to manage schemas and guarantee data compatibility. Luckily,
the Confluent Platform makes this easy and nearly transparent to the application
developer.

There are two ways to interact with Kafka: using a native client for your
language combined with serializers compatible with the the Schema Registry, or
using the Kafka REST Proxy. Most commonly you will use the serializers if your
application is developed in a language with supported serializers, whereas you
would use the REST Proxy for applications written in other languages.

Java applications can use the standard Kafka producers and consumers, but will
substitute the default ByteArraySerializer with
io.confluent.kafka.serializers.KafkaAvroSerializer (and the equivalent
deserializer), allowing Avro data to be passed into the producer directly and
allowing the consumer to deserialize and return Avro data.

For a Maven project, include dependencies within the dependencies tag for
the Avro serializer and for your desired version of Kafka:

In your code, you can create Kafka producers and consumers just as you normally
would, with two adjustments:

The generic types for key and value should be Object. This allows you to
pass in primitive types, Maps, and Records.

Set the key/value serializer or deserializer and Schema Registry URL
options. For example if you were configuring a producer directly in your code:

Propertiesprops=newProperties();props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,io.confluent.kafka.serializers.KafkaAvroSerializer.class);props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,io.confluent.kafka.serializers.KafkaAvroSerializer.class);props.put("schema.registry.url","http://localhost:8081");// Set any other propertiesKafkaProducerproducer=newKafkaProducer(props);

We recommend these values be set using a properties file that your
application loads and passes to the producer constructor. The settings are
similar for the deserializer.

Now your application code can send Avro data that will be automatically
serialized and the schema will be registered or validated against the Schema
Registry. The application code is essentially the same as using Kafka without
the Confluent Platform:

C and C++ applications can use the librdkafka client with libserdes for Avro and
Schema Registry support. The Kafka Clients section discusses general
usage of librdkafka for producing and consuming data in any format. In this
section we cover how to integrate libserdes to serialize Avro and track schemas
in the Schema Registry.

First add the libserdes dependency to your project by including the headers and
linking to the library. This process will vary by build system. Here is an
example of how we can get the relevant values into Makefile variables:

When consuming messages, you can deserialize an Avro message that was produced
with the above code. This looks up the schema in the Schema Registry
automatically if it is not already cached locally and then deserializes the
data:

The REST Proxy should be used for any language that does not have native clients
with serializers compatible with the Schema Registry. It is a convenient,
language-agnostic method for interacting with Kafka. Almost all standard
libraries have good support for HTTP and JSON, so even if a wrapper of the API
does not exist for your language it should still be easy to use the API. It also
automatically translates between Avro and JSON. This simplifies writing
applications in languages that do not have good Avro support.

The REST Proxy API Reference describes the complete API in
detail, but we will highlight some key interactions here. First, you will want
to produce data to Kafka. To do so, construct a POST request to the
/topics/{topicName} resource including the schema for the data (plain
integers in this example) and a list of records, optionally including the
partition for each record.

Note that REST Proxy relies on content type information to properly convert data
to Avro, so you must specify the Content-Type header. The response
includes the same information you would receive from the Java clients API about
the partition and offset of the published data (or errors in case of
failure). Additionally, it includes the schema IDs it registered or looked up in
the Schema Registry.

In future requests, you can use this schema ID instead of the full schema,
reducing the overhead for each request. You can also produce data to specific
partitions using a similar request format with the
/topics/{topicName}/partitions/{partition} endpoint.

To achieve good throughput, it is important to batch your produce requests so
that each HTTP request contains many records. Depending on durability and
latency requirements, this can be as simple as maintaining a queue of records
and only send a request when the queue has reached a certain size or a timeout is
triggered.

Consuming data is a bit more complex because consumers are stateful. However, it
still only requires two API calls to get started. See the
API Reference for complete details and examples.

Finally, the API also provides metadata about the cluster, such as the set brokers, list
of topics, and per-partition information. However, most applications will not
need to use these endpoints.

Note that it is also possible to use
non-Java clients
developed by the community and manage registration and schema validation manually using the
Schema Registry API. However, as this is error-prone
and must be duplicated across every application, we recommend using the REST
Proxy unless you need features that are not exposed via the REST Proxy.