Scaling Telemetry Systems with Streaming

Streaming systems are essential to doing large-scale data analysis and providing real-time insights into data. In this talk, members of Honeycomb's platform engineering team will describe the evolution of our Kafka cluster and producers/consumers over the past 4 years as incoming telemetry volume increased by more than 10x. We'll discuss how we made choices on the broker/server side around instance types and sizing, tradeoffs of local SSD speed vs durability, optimizing network traffic, and doing chaos engineering on brokers to ensure we could survive loss of instances. We'll cover patterns for distributing data between partitions, scheduling updates of consumers, observing results of changes to cluster configuration and more.