论文精选73°

文本误导下的音频对话推理研究

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

精选理由

ContraTalk揭示了语音理解中的文本偏差问题,Audio Twin框架为多模态推理提供了新思路。

AI 摘要

研究人员提出ContraTalk基准测试,包含501个问题,涵盖交互行为、情感状态等五个话语维度。实验显示,纯文本LLM在一致案例中准确率超90%,但在冲突案例中降至33-48%。Audio Twin框架通过将语音转换为文本可读表示,改善了冲突案例的准确率并减少了陷阱选择。

原文 · arXiv cs.AI

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

Understanding spoken dialogue requires joint reasoning over lexical content and paralinguistic acoustic signals such as emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and speech-grounded interpretations agree, enabling evaluation beyond adversarial audio dependence. This results in ContraTalk, a controlled benchmark containing 501 questions across five discourse dimensions: interaction behavior, emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving speech-grounded reasoning.