File Systems Unfit as Distributed Storage Back Ends: Lessons from 10 Years of Ceph Evolution

For a decade, the Ceph distributed file system followed the conventional wisdom of building its storage back end on top of local file systems. The experience with different file systems showed that this approach always leaves significant performance on the table while incurring significant accidental complexity. Therefore, the Ceph team embarked on an ambitious project to build BlueStore, a new back end designed to run directly on raw storage devices. Somewhat surprisingly, BlueStore matured in less than two years. It outperformed back ends built atop file systems and got adopted by 70% of users in production.