HSPMD: Hierarchical and Heterogeneous SPMD for Distributed DL Training

Haoyang Li, Peking University; Fangcheng Fu, SJTU; Hao Ge, Sheng Lin, Xuanyu Wang, Jiawen Niu, and Yuming Zhou, Peking University; Xupeng Miao, Purdue University; Bin Cui, Peking University