Madaari: Ordering for the Monkeys

Wednesday, March 27, 2019 - 12:20 pm12:50 pm

Ashutosh Raina and Ramprasad Ellupuru, eBay


Lineage Driven Fault Injection (LDFI) is a state of the art technique in chaos engineering experiment selection. As SRE's we would like to perform chaos experiments that reveal the bugs that the customers are most likely to hit first. In this talk, we present new improvements to LDFI that orders the experiment suggestions.

In the first the half of the talk we will show introduce LDFI as a technique that can be widely used within an enterprise. We also highlight how ordering is general purpose technique that we can use to encode the peculiarities of a heterogeneous microservices architecture. LDFI can work in an enterprise by harnessing the observability infrastructure to model the redundancy of the system.

Next, we present experiments conducted within eBay using ordered LDFI and some preliminary results. We show examples of services where we discovered bugs, and how carefully controlling the order of experiments allowed LDFI to avoid running unnecessary experiments.

We will discuss open problems and future direction of LDFI.

Key takeaways :

  1. Understand how LDFI can be integrated in the enterprise by harnessing the observability infrastructure
  2. Limitations of LDFI w.r.t unordered solutions and why ordering matters for chaos experiments
  3. Preliminary results of prioritized LDFI and a future direction for the community

No prior knowledge of LDFI is required.

Ashutosh Raina, eBay

Ashutosh is a member of the Site Reliability team at eBay focussed on bringing LDFI to the enterprise. He works at the intersection of academia and industry, trying his best to fuse them together. Previously, Ashutosh was a graduate student at UCSC working at Disorderly Labs making distributed systems safer using LDFI.

Ramprasad Ellupuru, eBay

Ramprasad is a member of the Site Reliability team at eBay working on making checkout highly reliable and available. He is an experienced developer and a new practitioner of chaos engineering at eBay.

SREcon19 Americas Open Access Videos Sponsored by

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.

@conference {229503,
author = {Ashutosh Raina and Ramprasad Ellupuru},
title = {Madaari: Ordering for the Monkeys},
year = {2019},
address = {Brooklyn, NY},
publisher = {USENIX Association},
month = mar,

Presentation Video