How We Used Kafka to Scale Database Infrastructure

Friday, June 14, 2019 - 9:00 am10:00 am

Basavaiah Thambara, LinkedIn


This talk will introduce our home grown NoSQL document store and its initial design based on MySQL replication and the challenges faced, will touch base on some of the major changes incorporated in the product for replacing MySQL replication with Kafka-based replication and cover Kafka based replication implementation in some depth. Followed by an overview of Kafka configuration options to support reliable delivery along with details of the application logic to ensure "exactly once delivery" and "rejection of out-of-band messages."

Audience takeaways are

  • How our home grown NoSQL datastore works
  • Challenges we faced with MySQL based replication
  • How Kafka helped scale the database infrastructure
  • Over all how to use kafka for database replication at scale

Basavaiah Thambara, LinkedIn

Basavaiah Thambara (Basu) has more than a decade of experience designing, building, and scaling MySQL databases. He is currently working as a staff database engineer at LinkedIn managing Espresso, an in-house distributed NoSQL datastore built on top of MySQL. Prior to LinkedIn, he worked at Yahoo! after his Masters in Computer Science. He currently lives in Bangalore, India.

@conference {233315,
author = {Basavaiah Thambara},
title = {How We Used Kafka to Scale Database Infrastructure},
year = {2019},
address = {Singapore},
publisher = {{USENIX} Association},