ClinFusion: 面向整体医学理解的视觉中心多模态LLM系统

ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

精选理由

ClinFusion用新架构统一2D/3D医学图像,在20/24基准上碾压Hulu-Med、Lingshu,甚至超GPT-5.2。医生盲评说它报告最好,值得关注。

AI 摘要

ClinFusion提出组合级联视觉编码器架构Cascade Spatial-Aware Locality Fusion,统一2D和3D医学图像理解。引入MedIF-Bench进行指令跟随评估,并采用基于感兴趣区域的方法实现临床对齐的事实性报告生成评估。在24项基准中,ClinFusion以20项超越开源医学MLLM如Hulu-Med和Lingshu,并在13项上超过GPT-5.2和Gemini-3-Flash。放射科医生盲评确认ClinFusion生成最高排名报告,且其ROI基础指标与专家判断相关性最强。

原文 · arXiv cs.AI

ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.