Papers

OmniScore-Bench 发布:用 3000 道题评测 VLM 的乐谱阅读能力

OmniScore-Bench: Evaluating Vision-Language Models on Musical Score Reading

Why we picked it

看惯了文档问答和 OCR 基准,乐谱这块一直缺细粒度评测。这篇论文用 3000 道题测了 12 个 VLM,能拆出模型在音高识别、转谱、多页乐谱上的具体短板。做文档理解或多模态评测的团队可以直接拿来用。

一篇新论文提出 OmniScore-Bench,专门评测视觉语言模型(VLM)在乐谱识别上的光学音乐识别(OMR)能力。基准包含 3,000 道题,覆盖 6 个能力维度和 24 个任务子类型,每道题规定了可见范围、参考标注和答案格式,可以从符号定位一直测到音乐一致性推理和带证据链接的问答。研究团队对 12 个 VLM 进行了评测,发现不同模型在不同任务上的能力分布差异明显,摘录、单页和多页转谱任务的第一名各不相同。

Body

评测设计上有两个细节值得注意:一是控制变量分析,去掉乐谱图像后测量模型在音高和反复记号解读上对视觉输入的依赖程度,以及仅靠公开音乐背景知识还能剩多少分;二是把答案格式违规和真正的音乐错误分开统计,避免混淆两类失败。这解决了现有 OMR 数据集只做检测和转谱评分、无法定位 VLM 长处与短板的问题。

对做文档理解、多模态评测或音乐科技的开发者来说,这个基准提供了一套可自动评分的细粒度诊断框架,能定位模型在哪些乐谱操作上可靠、哪些需要改进。

Original · arXiv cs.AI

OmniScore-Bench: Evaluating Vision-Language Models on Musical Score Reading

Vision-language models (VLMs) can answer questions about musical scores, but their ability to recognise notation, recover its structure and interpret its musical meaning requires separate evaluation. Existing optical music recognition (OMR) resources establish detection and transcription measurements; a transcription score alone leaves the strengths and bottlenecks of a VLM score reader unresolved. We introduce OmniScore-Bench, an evaluation of OMR capabilities in VLMs through 3,000 questions spanning six aspects and 24 task subtypes. Each question specifies a visible scope, reference annotations and an answer schema, enabling automatic evaluation from symbol localisation to musical consistency and evidence-linked question answering. Our assessment of twelve VLMs reveals operation-specific capability profiles and different transcription leaders across excerpt, single-page and multipage tasks. Controlled analyses examine the roles of score images, auxiliary givens and additional page context. On the tested subset, image removal exposes visual dependence in pitch and repeat interpretation while measuring how much performance remains with public context alone. Separate format and content analyses distinguish answer-format violations from musical errors. OmniScore-Bench supports diagnosing which score-reading operations VLMs perform reliably and which require improvement.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include OmniScore-Bench, VLM, OMR, multimodal.

Source attribution: curated from arXiv cs.AI (@Congren Dai, Zhancheng Guo, Danni Zhao, Wenchu Wang, Xiaobing Li, Maosong Sun).