Kafka on Kubernetes: From Evaluation to Production at Intuit

Kubernetes is fast becoming the platform of choice for running distributed, containerized applications in the cloud. It has great features for availability, scalability, monitoring, ease of deployment, a rich set of tools and an extremely fast-growing ecosystem that is making it ever more useful.

However, running stateful applications such as Kafka on Kubernetes is not a common practice today. At Intuit, we took an experimentation and data driven approach for evaluating Kafka on Kubernetes in AWS. In this talk, we provide details of our requirements, the experimental setup, and details of the tests. Tests included basic functional tests for produce/consume of messages, network isolation tests, cross-region produce/consume, performance and scale tests. We’ll also talk about the various tools used during this testing. We focus on the problems we ran into and how we addressed them.

This talk will demonstrate a Kubernetes cluster running Kafka along with the details of how each component is configured. Specifically, the kafka and zookeeper statefulsets, the configmaps used for storing the server.properties used by all brokers, the service objects for enabling access to the brokers and, last but not least, integration with Splunk and Wavefront for logging and monitoring respectively.