I have a strange behavior with Kafka 0.8.1.1 where consumers do notretrieve the latest messages.

I created a topic called "my-topic" with 8 partitions. My applicationcreates a SimpleConsumer instance for each topic-partition pair andexecutes an OffsetRequest to retrieve the earliest and latest offsets foreach partition.

As an example, the partition 5 of the topic "my-topic" has an earliestoffset at 4 and a latest offset at 9.

Then, my application uses the same SimpleConsumer instance and executes aFetchRequest to read messages (starting from earliest) for that partition.

Still with the example with partition 5, my application can read messageswith offset 4,5,6,7,8 and then no more messages can be read. The samebehavior exists for other partitions.

I'm not sure that the message with offset "9" really exists, but the KafkaAPI returns a latest offset of 9 that let me thinking that this messageexists. I tested with a HighLevelConsumer and it works the same way,reading messages 4...8 but not the last one.

I did not notice this behavior with the previous version of Kafka I used(0.8.0-beta1).

This makes sense if you think about it. If you want to start "now" youdon't want the last message in the log, which could be seconds,minutes, days, or weeks old, you actually want the next message thatcomes in.