这篇论文让你搞懂语音语言模型内部是怎么偷偷把语音转成文本再推理的，分析得很透彻，适合想深入理解多模态模型原理的人。
该论文分析了来自不同模型族和规模的交错语音文本语言模型，发现它们会在中间层隐式转录语音对应的文本词，其中77%的数据中该文本词出现在Top候选词中。随后模型在文本空间预测下一个词，再转回语音域。研究还表明，交错训练数据和文本LM初始化是诱发该行为的关键，且该行为与口语知识能力相关。
Interleaved Speech Language Models Latently Work In Text
Speech language models (SLMs) have been extensively studied, with the common paradigm incorporating text data and pre-trained text LMs. A leading approach is speech-text interleaving in which models are trained over sequences containing both speech and text tokens, aiming to boost even speech-only capabilities. Yet the way these two modalities interact in the model latent space remains unclear. In this work, we analyze interleaved speech-text LMs from different model families and sizes through the scope of the logit lens to provide such insight. We reveal that these models go through an implicit transcription phase in which the text token of the spoken word becomes decodable in intermediate layers, despite not being trained for speech recognition. The transcription of the word appears as one of the top candidate words for as much as 77\% of the data. Following this stage, the models proceed to predict the next word in the text space before transforming back to the speech domain. We finally analyze the role of interleaving data, and initializing from text LMs in eliciting this behavior, as well as seeing how this correlates with spoken knowledge abilities. Our analysis sheds light on the internal mechanisms underlying the relationship between speech and text modalities and could shape SLM optimization.