UbiHearo 论文提出面向听障用户的场景感知声音辅助移动智能体
UbiHearo: Bringing Scenario-Aware Sound Guidance to DHH Users with Mobile Agents
一篇把声音语义讲清楚的听障辅助论文,端侧音频语言模型靠结构化中间表示推理,不用写提示词,省电还能个性化。做端侧多模态、无障碍应用的研究者和工程师值得细看,车载场景的实验也有参考价值。
arXiv 论文 UbiHearo 针对听障(DHH)用户的声音感知问题,提出一个能闭环完成感知—推理—行动的移动端听觉辅助智能体。现有移动助手大多把声音简化为孤立标签或笼统提醒,无法区分同一声音在不同场景下的含义与紧急程度。
技术上 UbiHearo 采用神经符号(neuro-symbolic)架构,把连续的声学与物理上下文流转化为结构化中间表示(IR)。三个核心机制包括:感知阈值触发以实现省电的常开监听并保留安全关键提示的确定性覆盖;分阶段神经符号推理框架让端侧音频语言模型基于结构化 IR 完成免提示的场景理解;以及有界人机协同个性化机制,通过轻量端侧决策边界更新适配用户偏好。
团队在智能手机和车载系统上跨多种场景评测,结果显示在精度—延迟—能耗上取得较优折衷,并支持个性化引导。对端侧多模态推理和无障碍应用开发者而言,这篇论文展示了结构化 IR 替代提示词工程的可行路径。
UbiHearo: Bringing Scenario-Aware Sound Guidance to DHH Users with Mobile Agents
For Deaf and Hard-of-Hearing (DHH) users, effective sound awareness goes beyond detecting \emph{what} sound occurs; it requires understanding \emph{what it means}, \emph{how urgent it is}, and \emph{what to do}. However, existing mobile assistants largely reduce acoustic scenes to isolated labels or generic alerts, failing to provide contextual guidance when the same sound implies different risks across scenarios. We present \sysname, a mobile hearing assistant agent that closes the sensing--reasoning--action loop for \textit{scenario-aware} and \textit{personalized} DHH assistance. \sysname adopts a \textit{neuro-symbolic} design that transforms continuous acoustic and physical context streams into structured intermediate representations (IRs) for controllable reasoning and guidance. It introduces three key mechanisms. \textit{First}, a perception-thresholded sensing mechanism enables energy-efficient always-on awareness by activating processing only for perceptually meaningful events while preserving safety-critical cues through deterministic overrides. \textit{Second}, a staged neuro-symbolic reasoning framework performs promptless scenario understanding by conditioning an on-device audio-language model on structured IRs instead of user-authored context descriptions. \textit{Third}, a bounded human-in-the-loop personalization mechanism adapts user-specific guidance through lightweight on-device decision-boundary updates while preserving population-aligned reasoning. We evaluate \sysname on smartphones and in-vehicle systems across diverse scenarios. It achieves favorable accuracy--latency--energy tradeoffs and personalized guidance.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include audio understanding, neurosymbolic AI, accessibility, edge computing.
Source attribution: curated from arXiv cs.AI (@Fengmin Wu, Sicong Liu, Zimu Zhou, Chenren Xu, Wanbing Zhao, Peiwen Sun, Zhiwen Yu).