Scaling Telemetry Systems with Streaming

Tuesday, March 21, 2023 - 11:00 am11:45 am

Liz Fong-Jones and Terra Field,


Streaming systems are essential to doing large-scale data analysis and providing real-time insights into data. In this talk, members of Honeycomb's platform engineering team will describe the evolution of our Kafka cluster and producers/consumers over the past 4 years as incoming telemetry volume increased by more than 10x. We'll discuss how we made choices on the broker/server side around instance types and sizing, tradeoffs of local SSD speed vs durability, optimizing network traffic, and doing chaos engineering on brokers to ensure we could survive loss of instances. We'll cover patterns for distributing data between partitions, scheduling updates of consumers, observing results of changes to cluster configuration and more.

Liz Fong-Jones,

Liz is a developer advocate, labor and ethics organizer, and Site Reliability Engineer (SRE) with 18+ years of experience. She is currently the Field CTO at Honeycomb, and previously was an SRE working on products ranging from the Google Cloud Load Balancer to Google Flights.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@conference {286307,
author = {Liz Fong-Jones and Terra Field},
title = {Scaling Telemetry Systems with Streaming},
year = {2023},
address = {Santa Clara, CA},
publisher = {USENIX Association},
month = mar

Presentation Video