Run migrations. This will create the model used to store Kafka log position offsets.:

$ python manage.py migrate logpipe

Usage

Serializers

The first step in either sending or receiving messages with logpipe is to define a serializer. Serializers for logpipe have a few rules:

1. Must be either a subclass of rest_framework.serializers.Serializer or a class implementing an interface that mimics rest_framework.serializers.Serializer.
1. Must have a MESSAGE_TYPE attribute defined on the class. The value should be a string that defines uniquely defines the data-type within it’s Topic / Stream.
2. Must have a VERSION attribute defined on the class. The value should be a monotonic integer representing the schema version number.
3. Must have a KEY_FIELD attribute defined on the class, representing the name of the field to use as the message key. The message key is used by Kafka when performing log compaction and by Kinesis as the shard partition key. The property can be omitted for topics which do not require a key.
4. If the serializer will be used for incoming-messages, it should implement class method lookup_instance(cls, **kwargs). This class method will be called with message data as keyword arguments directly before instantiating the serializer. It should lookup and return the related object (if one exists) so that it can be passed to the serializer’s instance argument during initialization. If no object exists yet (the message is representing a new object), it should return None.

Use the register_consumer decorator to register as many consumers and topics as you need to work with. Then, run the run_kafka_consumer command to process messages for all consumers automatically in a round-robin fashion.

$ python manage.py run_kafka_consumer

Dealing with Schema Changes

Schema changes are handled using the VERSION attribute required on every serializer class. When sending, a producer includes the schema version number in the message data. Then, when a consumer receives a message, it looks for a register serializer with a matching version number. If no serializer is found with a matching version number, a logpipe.exceptions.UnknownMessageVersionError exception is raised.

To perform a backwards-incompatible schema change, the following steps should be performed.

Update consumer code to have knowledge of the new schema version.

Update producer code to being sending the new schema version.

After some amount of time (when you are sure no old-version messages still exist in Kafka), remove the code related to the old schema version.

For example, if we wanted to require an email field on the Person model we defined above, the first step would be to update consumers to know about the new field.: