Yifan Dai, University of Wisconsin—Madison; Jacopo Tagliabue, Bauplan Labs; Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Tyler R. Caraza-Harter, University of Wisconsin—Madison
We present Kelvin, a DAG execution engine for single-machine data pipelines that approaches absolute zero-copy by enabling containerized stages to directly share physical memory through the in-memory file system. At the core of Kelvin is DeAnon, a lightweight Linux kernel module that converts anonymous memory pages into file-backed pages in place, allowing data produced by unmodified applications to become referable across processes without copying. Building on this capability, Kelvin extends Arrow-based communication to preserve sharing across pipeline transformations and incorporates runtime support to reuse deserialized inputs and manage shared memory across containers. Experiments show that Kelvin eliminates most intermediate copying, enabling pipeline stages to directly share physical memory; as a result, Kelvin improves pipeline throughput by up to 28× while reducing memory consumption and swapping.
Open Access Media
USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.
