InstaBind-Lite:量化视觉语言模型密集同类属性错绑

Diagnosing Dense Same-Class Attribute Misbinding in Large Vision-Language Models

精选理由

想测多模态模型在拥挤场景里会不会张冠李戴?这个新基准InstaBind-Lite能精准量化,开源模型错绑率近20%,比想象中严重。

AI 摘要

本研究提出密集同类属性错绑(DSCAM)问题,指视觉语言模型在拥挤场景中将属性错误分配给同类实例。作者发布InstaBind-Lite基准,含524张图像、529组3-6个同类实体、1773个框选实例及9580个确定性评估问题。测试5个开源和2个商业/API模型,开源系统平均错绑率19.84%,API系统7.55%,且80%以上错误源于相邻实例。该基准将错误答案分类为可识别来源,检验模型是否知道每个属性归属哪个实例。

原文 · arXiv cs.AI

Diagnosing Dense Same-Class Attribute Misbinding in Large Vision-Language Models

Large vision-language models can recognize the objects and attributes in a crowded scene yet assign an attribute to the wrong same-class instance. Generic visual-question-answering accuracy marks the response as wrong, while object-hallucination metrics may regard both the object and attribute as image-supported; neither reveals the transfer. This study formalizes this blind spot as Dense Same-Class Attribute Misbinding (DSCAM) and presents InstaBind-Lite, a controlled benchmark that makes it directly measurable. Its 524 images contain 529 curated groups of 3-6 same-class entities, 1773 boxed instances, ordered neighbors, distinguishable color-like attributes, and four complementary question levels, yielding 9580 deterministically evaluated questions. Unlike existing protocols, source-instance annotations separate unsupported generation and recognition failure from an attribute copied from another visible entity. Binding-specific metrics further quantify transfer frequency, adjacency, ordinal distance, and intervention effects. Across five open-source and two commercial/API models, the open-source systems average 19.84% Misbinding Rate and the API systems 7.55%; these errors are hidden by aggregate accuracy. Among identifiable transfers, 80.70% and 81.51%, respectively, originate from adjacent instances. Localization and instance-first interventions help selected models but are not universal remedies. InstaBind-Lite therefore turns previously undifferentiated wrong answers into source-identifiable failure categories and tests a reliability dimension that conventional benchmarks cannot determine: whether a model knows not only what is visible, but which instance owns each attribute.