Shopify's Kafka Producer Pipeline for Ruby on Rails

In the early fall our infrastructure team was considering Kafka , a highly available message bus. We were looking to solve several infrastructure problems that had come up around that time. We were looking for a reliable way to collect event data and send it to our data warehouse. We were considering a more service-oriented architecture, and needed a standardized way of message passing between the components. We were starting to evaluate containerization of Shopify, and were searching for a way to get logs out of containers. We were intrigued by Kafka due to its highly available design. However, Kafka runs on the JVM, and its primary user, LinkedIn, runs a full JVM stack. Shopify is mainly Ruby on Rails and Go, so we had to figure out how to integrate Kafka into our infrastructure. Avoiding a direct link between Ruby and Kafka The straight forward solution is to emit event...