想提升音频模型对语气、情感等非语义信息的识别?试试IAAN,在不重训模型的情况下,通过对编码器内部特定神经元做微调就能显著提分,多模型实测有效。
IAAN(Identifying and Amplifying Acoustic Neurons)是一种无需训练和标签的方法,通过对比真实波形与噪声参考的激活值,对音频编码器中的每个前馈神经元进行评分并放大得分最高的少量神经元。在10种非语义语音属性上,IAAN在Audio-Flamingo-3上平均准确率提升25.7个百分点,在Qwen2.5-Omni上提升21.4,在Kimi-Audio上提升9.7。实验表明,仅在编码器内部以神经元级粒度干预才能产生增益,而解码侧或语言模型内部干预效果微弱甚至退化。该方法还能进一步改善已经专门微调用于强调声学证据的模型。
Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models
Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content. Improving this without the cost of retraining calls for an effective inference-time intervention, yet most existing methods intervene only after the audio encoder and operate at a relatively coarse granularity. The encoder itself, where acoustic information is first extracted from the waveform, remains largely unexplored, especially at the level of individual neurons. We introduce IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information. IAAN then amplifies a small set of the highest-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. It also improves a model already explicitly fine-tuned to prioritize acoustic evidence. In controlled comparisons, both the encoder locus and neuron-level selectivity prove necessary for this gain. Intervening after the encoder, at the decoding side or inside the language model, yields little to no improvement, or even deteriorates accuracy. The improvement also depends on which specific neurons are amplified, not merely on their number, confirming that IAAN's acoustic score succeeds in identifying the neurons that matter. These results show that a small, precisely targeted intervention inside the audio encoder is an effective and largely untapped way to strengthen the acoustic understanding of LALMs, opening a new direction for inference-time methods that improve acoustic perception through neuron-level access to the encoder.