Papers

FG-GraphBench 基准发布:用多模态场景图评测细粒度视觉理解

FG-GraphBench: Benchmarking and Enhancing Fine-Grained Visual Understanding with Multimodal Scene Graphs

Why we picked it

搞多模态的研究者可以关注,这篇同时给了基准和方法:FG-GraphBench 拆成三个任务测细粒度理解,FiGRA 把场景图编码成图 token 喂进 MLLM,三个任务都跑赢了强基线。

一项新研究提出了 FG-GraphBench(Fine-Grained Multimodal Scene Graph Benchmark),一个针对多模态大语言模型细粒度视觉理解能力的基准,包含三个任务:细粒度视觉问答(FG-QA)、图像描述生成(FG-Caption)和图像描述(FG-Description)。基准采用两阶段构建流程,先发现并定位细粒度视觉实体,再建立实体间关系,生成多模态场景图(MSGs),以节点表示实体、边表示关系来组织视觉信息。研究指出两个核心问题:现有场景图基准对细粒度实体及其关系的覆盖不足,且现有方法未能在 MLLM 中有效利用图结构信息。

Body

针对这些问题,作者提出 FiGRA(Fine-grained Graph Representation Augmentation)方法,采用两阶段训练策略,将多模态实体和关系特征编码为连续的图 token 并注入 MLLM。实验结果显示,FiGRA 在 FG-GraphBench 的三个任务上均显著优于强基线模型,进一步分析证实了保留细粒度实体和关系信息对视觉理解的重要性。

Original · arXiv cs.AI

FG-GraphBench: Benchmarking and Enhancing Fine-Grained Visual Understanding with Multimodal Scene Graphs

Fine-grained visual understanding requires multimodal large language models (MLLMs) to recognize small visual entities and understand their relationships. Scene graphs offer a natural way to represent this information, organizing visual entities as nodes and their relationships as edges. However, we identify two challenges: existing scene graph benchmarks provide insufficient coverage of fine-grained entities and the relationships among them, and existing methods for fine-grained visual understanding have yet to effectively leverage such graph-structured information in MLLMs. To address these challenges, we introduce Fine-Grained Multimodal Scene Graph Benchmark (FG-GraphBench) and FiGRA. FG-GraphBench is a benchmark comprising three complementary tasks: fine-grained visual question answering (FG-QA), image captioning (FG-Caption), and image description generation (FG-Description). We develop a two-stage construction pipeline that discovers and grounds fine-grained entities and establishes their relations to construct the fine-grained multimodal scene graphs (MSGs) in FG-GraphBench. Building on these MSGs, we propose FiGRA, a Fine-grained Graph Representation Augmentation method for MLLMs. FiGRA adopts a two-stage training strategy. It encodes multimodal entity and relation features as continuous graph tokens, and integrates them into the MLLM. Experiments show that FiGRA significantly outperforms strong baselines across three tasks of FG-GraphBench. Further analyses confirm the effectiveness of FG-GraphBench and highlight the importance of preserving fine-grained entity and relation information for visual understanding.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include FG-GraphBench, FiGRA, multimodal, MLLM.

Source attribution: curated from arXiv cs.AI (@Sicheng Liu, Xunkai Li, Ziyu Han, Xingnan Zhang, Ru Zhang, Rong-Hua Li).