Lessons from Operating a Large-Scale Distributed Database with a Replicated Key-Value Engine

Jung-Sang Ahn, Navaneetha Krishnan Thanka Nadar, and Sami Ben-Romdhane, eBay Inc.

Modern distributed databases frequently rely on key-value storage engines as durable backends for replicated state machines. While prior work extensively studies storage engine internals, far less attention has been given to the system-level constraints that arise when such engines are deployed at scale under strict resource limits. In production, storage design directly impacts tail latency, availability, recovery time, and multi-tenant stability.

This paper presents operational insights from running a log-structured merge-tree (LSM-tree)-derived key-value engine as the storage foundation of eBay's cloud-native distributed database, which serves trillions of requests per day. We describe practical challenges and operational responses in snapshot binding and installation, scan overhead caused by visibility checks and predicate evaluation, CPU-constrained compaction, write amplification and noisy-neighbor effects in multi-tenant environments, and programmable compaction for unified maintenance. These experiences show that when the storage layer underpins a distributed database, priorities shift from peak throughput to latency stability, predictable recovery, and bounded background work. Some mechanisms in our system rely on properties of our storage engine, but the underlying integration lessons are more broadly applicable to replicated database deployments.

Category: 
Deployed-Systems Paper

Open Access Media

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.