这篇论文给VLM配了三种看图工具(表格、图表、放大),再用GRPO训练工具使用策略,比普通prompt和RL方法更准,适合处理科学图表验证。
多模态科学声明验证需依托论文中的图表、表格等视觉证据。现有方法在定位关键视觉证据、准确读取结构化科学图表、整合多模态观察方面存在不足。本文提出首个工具增强框架ToolSciVer,为视觉语言模型(VLM)配备三种类型感知视觉工具:表格行列聚焦、图表结构化解析、高分辨率区域放大。采用分组相对策略优化(GRPO)训练策略,综合奖励答案正确性、格式、长度、工具使用效率及有效性。在SciVer和MuSciClaims数据集上基于Qwen、InternVL、Gemma三个系列的五个VLM进行实验,结果表明该方法优于包括提示词和强化学习方法在内的四个基线。
ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning
Multimodal Scientific Claim Verification (MSCV) requires models to verify scientific claims using visually grounded evidence from papers, including figures, tables, charts, and textual context. However, existing methods often fail because they struggle to locate decisive visual evidence, accurately read structured scientific visuals, and integrate multimodal observations into reliable reasoning. We introduce ToolSciVer, the first tool-augmented framework for MSCV to our knowledge. ToolSciVer equips a VLM with three type-aware visual tools, table row/column focus, chart-to-structure parsing, and high-resolution region zoom, which convert dense scientific visuals into explicit, claim-facing evidence, and trains the policy with Group Relative Policy Optimization (GRPO) under a composite reward of answer correctness, format validity, length control, tool-use efficiency, and tool-validity penalties. Experiments on SciVer and MuSciClaims datasets on five VLMs from three model families (Qwen, InternVL, Gemma) demonstrate that our method achieves superior performance compared to four competitive baselines including prompting-based and RL-based tool-use methods, highlighting the effectiveness of learned, type-aware tool use for scientific claim verification.