SDCs in the Wild: Characterizing and Diagnosing SDC-defective GPUs in Production LLM Training

Wenxin Zheng, Shanghai Jiao Tong University, ByteDance Seed; Bin Xu, Jinyu Gu, and Xingda Wei, Shanghai Jiao Tong University; Haibo Chen, Shanghai JiaoTong University; Wenxiao Wang, Yun Zhang, Zuquan Song, Gaohong Liu, Yucheng Nie, Mingcong Han, Zhe Nan, Zhuolin Zheng, Huan Yu, Shuguang Wang, Ziming Zhou, Hang Zhu, and Wencong Xiao, ByteDance Seed; Xin Liu, ByteDance Inc.