Chaos-Driven Development: TDD for Distributed Systems

Thursday, March 23, 2023 - 11:30 am11:50 am

Dhishan Amaranath and Tucker Vento, Bloomberg LP


Chaos experimentation is a force multiplier for other Reliability Engineering practices. By taking the time to design an intentional suite of experiments early in the software development life cycle, we are able to more easily validate improvements and changes to the system as they are made. The broad applicability of Chaos Engineering allows us to apply test-driven development (TDD) principles to all manners of changes in a distributed system. In this talk, we will show how early adoption of these testing processes enables us to continually validate our expectations of a system as it grows. These principles can be applied to any distributed system, something we will illustrate through stories from our own experiences with both private and public cloud development.

Dhishan Amaranath, Bloomberg LP

Dhishan Amaranath works as a senior cloud SRE at Bloomberg, where he is building foundational elements on the public cloud. He is diligent about trying to apply all aspects of SDLC to anything he works on, and is especially passionate about all forms of testing and building resilient and reliable architectures. He also promotes the practice of Chaos Engineering across the firm through the company's Reliability Engineering Guild. When he is not geeking out, he immerses himself in listening to podcasts and books about behavioral economics. So, grab him for coffee and discuss why aisles at Target are always stocked the same, while Costco regularly moves stuff around.

Tucker Vento, Bloomberg LP

Tucker Vento is the team lead and product owner for Bloomberg's Resilience Engineering team, which is responsible for maintaining a Chaos Engineering platform and helping engineers run chaos experiments in Bloomberg's data centers. While attending another USENIX conference as an SRE, he learned about the value (and fun) of deliberately breaking distributed systems and subsequently embarked on a mission to make Bloomberg's engineers embrace the power of failure. When he is not busy learning how to break something, Tucker enjoys listening to fast cars, making bad music, and relaxing with his two cats.

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@conference {286232,
author = {Dhishan Amaranath and Tucker Vento},
title = {{Chaos-Driven} Development: {TDD} for Distributed Systems},
year = {2023},
address = {Santa Clara, CA},
publisher = {USENIX Association},
month = mar

Presentation Video