论文精选

MTRefSeg-21K:多时序指代分割开源基准与基线模型

An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation

精选理由

做多模态视觉推理或时序变化检测的团队,这个新基准和基线模型值得关注——它填补了LVLM在时序推理上的空白,可以直接用来评估和提升模型能力。

AI 摘要

多时序指代分割(MTRS)是一项新任务,要求模型从多时序图像中分割出语言描述的时间变化。研究团队提出了CRAFT-Agent自动化数据构建流程,并构建了首个基准MTRefSeg-21K,包含2.1万高质量三元组。现有视觉语言模型在该任务上表现不佳,为此团队提出了MTRefSeg-R1框架,采用两阶段训练策略,先学习通用时序变化感知,再微调实现细粒度语言引导定位。实验表明,MTRefSeg-R1显著优于现有基线,揭示了多时序视觉推理的挑战与潜力。

原文 · arXiv cs.AI

An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation

Large Vision-Language Models (LVLMs) have shown strong visual understanding and language-guided grounding abilities, yet their capacity for multi-temporal visual reasoning remains underexplored. To bridge this gap, we introduce \textbf{Multi-temporal Referring Segmentation (MTRS)}, a new task that aims to segment language-described temporal changes from multi-temporal images. MTRS extends conventional referring segmentation and change detection by jointly requiring temporal correspondence reasoning, language grounding, and pixel-level mask prediction. We propose \textbf{CRAFT-Agent}, an automated data construction pipeline with human auditing, and build \textbf{MTRefSeg-21K}, the first MTRS benchmark, containing 21K high-quality multi-temporal image-text-mask triplets across diverse scenes, viewpoints, and domains. Benchmarking a broad set of VLM- and LVLM-based models reveals that direct inference performs poorly, while task-specific fine-tuning remains limited. To address this, we propose \textbf{MTRefSeg-R1}, a change-aware LVLM framework trained with a two-stage strategy. It first learns general temporal-change perception from 20K vision-only bi-temporal samples, and is then fine-tuned on MTRefSeg-21K for fine-grained language-guided temporal localization. MTRefSeg-R1 explicitly models cross-temporal visual differences, aligns language instructions with temporal variations, and predicts referred change masks. Extensive experiments show that MTRefSeg-R1 achieves strong and often superior performance compared with existing LVLM baselines, demonstrating the challenge and potential of MTRS.