When Segment Met Cricket: A Scaling Story

Segment receives over a half-million events per second from web sites, mobile applications, and other data sources around the world. Last summer, an unexpectedly popular event pushed our AWS-and-Kafka-based event pipeline to its limits and tested our ability to scale quickly to handle the crushing demand. Yet through access to crucial data and quick thinking, we were able to avoid catastrophe. We’ll discuss in detail the events that led to the crisis, assumptions we made that proved to be false, the key observations we had that helped us rescue the system before it was too late, and how we gained a tremendous increase in capacity for a small amount of additional investment.