GEM: GPU Erasure-Coded Memory for Fast Training Recovery

Training large deep learning models typically requires a massive number of GPUs operating continuously for weeks or even months. Unfortunately, GPU memory failures are not uncommon in such long-running jobs. When these failures occur, existing solutions reload the latest checkpoint from CPU memory or remote persistent storage to resume training, incurring substantial storage overheads and expensive network transfers.

This paper presents GEM, a lightweight fault-tolerant mechanism that introduces GPU Erasure-coded Memory for fast recovery from GPU memory failures. GEM first employs selective erasure coding (EC) to protect critical model states instead of relying on periodic checkpointing. Our insight is that these critical states (including parameters and optimizer states) are updated infrequently, allowing them to be protected with EC in a computation-/storage-efficient manner. GEM adopts a wide-stripe EC design to minimize recovery redundancy, enabling redundancy to reside entirely in GPU memory and avoiding additional data transfers. Furthermore, GEM incorporates a mixed-granularity retry mechanism to recover the remaining frequently updated data. We conduct extensive experiments on NVIDIA GPUs with real-world deep learning models and show that GEM reduces wasted time caused by GPU memory failures by an average of 96.6%, while incurring only 0.3% additional GPU memory overhead.