论文精选

Vision-OPD:通过自蒸馏让多模态大模型看清细节

Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation

精选理由

多模态模型开发者常头疼的“看不清细节”问题,Vision-OPD用自蒸馏给出了一个轻量解法——不用外部模型或标注,直接让模型学会“自动放大”关键区域。做细粒度视觉理解或MLLM优化的团队值得关注。

AI 摘要

多模态大模型在细粒度视觉理解任务中常因无法聚焦关键证据而失败,而非缺乏局部识别能力。研究者提出Vision-OPD框架,通过区域到全局的自蒸馏方法,让模型从裁剪后的局部图像(教师)中学习,并迁移到全图(学生)策略上。该方法无需外部教师模型、标注数据或推理时工具,仅通过最小化教师与学生间token级分布差异来提升性能。在多个细粒度视觉理解基准上,Vision-OPD模型性能优于或媲美更大规模的开源、闭源及“思考+图像”智能体模型。

原文 · arXiv cs.AI

Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation

Multimodal Large Language Models (MLLMs) still struggle with fine-grained visual understanding, where answers often depend on small but decisive evidence in the full image. We observe a regional-to-global perception gap: the same MLLM answers fine-grained questions more accurately when conditioned on evidence-centered crops than on the corresponding full images, suggesting that many failures stem from difficulty to focus on relevant evidence rather than insufficient local recognition ability. Motivated by this observation, we propose Vision-OPD (Vision On-Policy Distillation), a regional-to-global self-distillation framework that transfers the model's own privileged regional perception to its full-image policy. Vision-OPD instantiates two conditional policies from the same MLLM: a crop-conditioned teacher and a full-image-conditioned student. The student generates on-policy rollouts, and Vision-OPD minimizes token-level divergence between the teacher and student next-token distributions along these rollouts. This enables the model to internalize the benefit of visual zooming without external teacher models, ground-truth labels, reward verifiers, or inference-time tool use. Experiments on multiple fine-grained visual understanding benchmarks show that Vision-OPD models achieve competitive or superior performance against much larger open-source, closed-source, and "Thinking-with-Images" agentic models.