Airbnb scaled Spark streaming with a novel balanced Kafka reader that can ingest massive amount of logging events from Kafka in near real-time. This post summarizes the challenges they faced and the selected solution.