Kafka on Kubernetes: From Evaluation to Production at Intuit

Tuesday, October 30, 2018 - 2:00 pm2:30 pm

Shrinand Javadekar, Intuit, Inc.

Abstract: 

Kubernetes is fast becoming the platform of choice for running distributed, containerized applications in the cloud. It has great features for availability, scalability, monitoring, ease of deployment, a rich set of tools and an extremely fast-growing ecosystem that is making it ever more useful.

However, running stateful applications such as Kafka on Kubernetes is not a common practice today. At Intuit, we took an experimentation and data driven approach for evaluating Kafka on Kubernetes in AWS. In this talk, we provide details of our requirements, the experimental setup, and details of the tests. Tests included basic functional tests for produce/consume of messages, network isolation tests, cross-region produce/consume, performance and scale tests. We’ll also talk about the various tools used during this testing. We focus on the problems we ran into and how we addressed them.

This talk will demonstrate a Kubernetes cluster running Kafka along with the details of how each component is configured. Specifically, the kafka and zookeeper statefulsets, the configmaps used for storing the server.properties used by all brokers, the service objects for enabling access to the brokers and, last but not least, integration with Splunk and Wavefront for logging and monitoring respectively.

Shrinand Javadekar, Intuit, Inc.

Shrinand Javadekar is a "Kubernetes junkie" in the Platform team at Intuit. His primary focus is to make simple, scalable and reliable Kubernetes clusters, the de-facto platform for developing, deploying and running apps. In the past he has been part of large scale file system and virtualization projects at EMC and VMWare. However, his most fun stints have been working on cloud-native platforms and services at startups such as Maginatics and Applatix!

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

BibTeX
@conference {221722,
author = {Shrinand Javadekar},
title = {Kafka on Kubernetes: From Evaluation to Production at Intuit},
year = {2018},
address = {Nashville, TN},
publisher = {{USENIX} Association},
month = oct,
}

Presentation Video 

Presentation Audio