研究显示语音语言模型对同一音位特征的听觉与文字表征并不一致
Do Audio Language Models Hear and Read Distinctive Features Alike?
arXiv 上的新论文，测了 6 个音频模型怎么在内部表征音位，发现只有 Qwen2.5-Omni 在浊音上真把听和读对齐了，分析挺细。
一篇 arXiv 论文测试音频语言模型在听到和读到同一音位特征时，decoder 内的表征方向是否一致。方法是对只差一个特征的最小对立音位对取表征均值偏移，再计算语音流与文字流之间的余弦相似度，并与随机配对构建的参照基线比较，而非与零比较。实验覆盖 6 个模型、7 个特征、来自 11 个语系的 15 种语言。经多重检验校正后，只有两个 Qwen2.5-Omni 模型在浊音特征上超过参照基线，且参照基线在不同模型之间相差达 7 倍。6 个模型中有 3 个在 14 种可测语言的语音侧共享同一个浊音表征方向，模型家族而非模型规模决定了表征方式。
Do Audio Language Models Hear and Read Distinctive Features Alike?
Audio language models pass speech and text through a single decoder. We ask whether that decoder represents a distinctive feature in the same direction when a phoneme is heard and when it is read. For minimal pairs of phonemes differing in one feature, we take the offset between the two members' mean representations. Averaging those offsets gives a direction for each stream, and we measure the cosine between the two. Because the two streams already agree about arbitrary phoneme pairs, we compare every measure against a reference built from random pairings rather than against zero. We apply this to 6 models, 7 features and 15 languages from 11 families. Only voicing in the two Qwen2.5-Omni models exceeds that reference after correction for multiple testing, and the reference varies by a factor of seven between models. In three of the six models, voicing has one direction in audio across the 14 languages with enough minimal pairs to measure it, and every language pair agrees in two of them. The model family, not the model size, predicts which stream represents a feature.