In the past couple of months, I've put quite some work into the PHP client library for Apache Kafka, a persistent, distributed, high-throughput publish-subscribe messaging system. Over a year ago I sort of reverse-engineered the protocol, and published a first rudimental library to produce and consume messages with version 0.05. Since then, the Kafka project has evolved a lot, moving from the LinkedIN repositories to the Apache Incubator, gaining traction, committers, features. The protocol has changed slightly too, so the old library doesn't work anymore with the new version (0.6+).

New PHP client library

The new library fixes many of the shortcomings of the previous version, and adds tons of functionality:

added support for GZIP compression (both in the producer and the consumer);

implemented stream-based iteration of messages (rather than loading the entire response in memory);

completely refactored socket handling, to be more robust, with better error checking and handling of edge-cases;

added support for 64bit offsets;

better checks for responses from Kafka (fixed connection hanging);

added Zookeeper-based consumer, with support for multiple consumer groups, and for manual offset commit action (so it's possible to wait for an ACK from the message processor before advancing the offsets), and example code;

added support for OffsetRequest and getOffsetsBefore() in the SimpleConsumer class to query the state of the queue and recover consumption after old offsets become invalid because of expired data;

support for connection timeouts in microseconds;

vastly improved test suite.

Overall, this is a pretty solid release, we've been using it in production for over a month without any problem.

Please note that in the Zookeeper-based consumer, the offsets are committed manually. This is a difference from the off-the-shelf Scala ZookeeperConsumer, as we needed to control when the offsets were committed, i.e. we wanted to commit the offset only after receiving an ACK that the messages were correctly processed.

Have fun

If you play with this code, please let me know about your setup and use-case, I'd love to get feedback and suggestions.