One Bad Token Spoils the Barrel: Assessment, Detection, and Remediation of Glitch Tokens in Large Language Models

Kunsheng Tang and Peigui Qi, University of Science and Technology of China; Yide Song, University of Washington; Wenbo Zhou, University of Science and Technology of China; Zhicong Huang, Ant Group; Qing Guo, Nankai University; Tianwei Zhang, Nanyang Technological University; Weiming Zhang and Nenghai Yu, University of Science and Technology of China; Jie Zhang, CFAR, A*STAR, Singapore