fs123: An Open-Source, Network Filesystem with Pervasive Caching

Monday, October 28, 2019 - 11:00 am11:45 am

Michael Fenn, D. E. Shaw Research, LLC


fs123 is a read-only, scalable, high-performance, network filesystem running in production, delivering petabytes of simulation data and tens of thousands of software packages to thousands of geographically-distributed clients.

The fs123 protocol is layered over HTTP and leverages that ecosystem (load balancers, proxies, redirects, etc.). The fs123 protocol provides mechanisms for loosely-coupled servers to assert that two files are the same to the client, which allows horizontal scaling of fs123 servers.

fs123 is WAN-friendly, requiring a minimum number of round-trips for each operation as a result of design decisions that require the client to make a minimum number of round trips and tunable caching features.

Since 2016, fs123 has run in production on ~5000 machines across several sites. The client uses the FUSE low-level API and can work through network outages (or even offline) once the on-disk cache is primed. The libevent-based server easily delivers data at 40 Gbps.

Michael Fenn, D. E. Shaw Research, LLC

Michael Fenn is a Research Engineer at D. E. Shaw Research, LLC and holds a B. S. and M. S. in Computer Science from Clemson University. He first became interested in fault-tolerant filesystems after suffering through one-too-many NFS outages. When he's not at work, he enjoys automotive track events and doing burnouts.

@conference {240820,
author = {Michael Fenn},
title = {fs123: An {Open-Source}, Network Filesystem with Pervasive Caching},
year = {2019},
address = {Portland, OR},
publisher = {USENIX Association},
month = oct,