论文72°

Audio-Interaction:统一流式音频交互的在线大语言模型

Audio Interaction Model

精选理由

做语音交互或智能助手的团队终于有了一个能同时处理流式对话、环境音和实时指令的统一模型,比拼凑多个专用模型高效太多,做音频 AI 的建议点开看看框架和数据集。

AI 摘要

现有大型音频语言模型(LALM)多为离线处理,而流式音频模型又各自为政,只能处理单一任务(如流式语音识别或语音聊天)。研究者提出 Audio Interaction Model 概念,通过始终在线的“感知-决策-响应”循环,实现实时听音、理解环境并即时反应。他们构建了 Audio-Interaction 统一流式模型,并配套 SoundFlow 框架(含流式数据构建、理解感知训练和异步低延迟推理)以及 260 万条流式语料 StreamAudio-2M。在 8 个基准测试中,该模型在保留主流音频任务性能的同时,解锁了实时 ASR、流式音频指令跟随和主动帮助等离线模型无法实现的能力。

原文 · arXiv cs.AI

Audio Interaction Model

Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting. It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly. We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following, from dialogue to full voice chatting, deciding when to respond from the semantics of the stream. To enable this, we propose SoundFlow, a framework that instantiates the perceive-decide-respond loop end to end, from data to training to deployment, through streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference for stable real-time interaction. We further construct StreamAudio-2M, a 2.6M-item streaming corpus spanning 7 fundamental abilities and 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Across 8 benchmarks, Audio-Interaction preserves competitive performance on mainstream audio tasks while unlocking capabilities inaccessible to offline LALMs, including real-time ASR, streaming audio instruction following, and proactive help.