Papers

3D 医学视觉语言模型经 CT-报告对比学习实现跨机构 HCC 分类

Efficient Self-Supervised Adaptation of 3D Abdominal Vision-Language Model for Institution-Specific HCC Classification via Full Fine-Tuning and PEFT

Why we picked it

医疗影像 VLM 微调的实操对比来了:同一模型做 HCC 分类,全量微调 AUC 0.82 但几个 epoch 就过拟合,ConvLoRA+LoRA 只训 1% 参数拿到 AUC 0.79。做医学 AI 落地或医院数据适配的团队可以对照自己的场景看取舍。

一项新研究将预训练的 3D 医学视觉语言模型(VLM)微调用于肝细胞癌(HCC)分类,训练数据来自与预训练语料获取特征不同的另一家机构的腹部 CT 与放射报告配对数据。论文对比了两种适配策略:全量微调取得 AUC 0.82、F1 0.71,PEFT 取得 AUC 0.79、F1 0.70。

Body

技术上,方法采用 CT-报告对比学习进行自监督适配,不需要病灶级标注或 HCC 分类标签作为训练目标。PEFT 方案在图像编码器使用 ConvLoRA、文本编码器使用 LoRA,仅更新约 1% 的模型参数即可获得接近全量微调的性能,但需要更多训练步数;全量微调收敛快,却在几个 epoch 内出现过拟合。研究还发现对比学习的温度初始化对分类性能影响显著。

结果显示该方法优于预训练基线,但仍低于有监督 HCC 分类方法的报告水平。对医疗 AI 落地而言,这项工作给出了跨机构域偏移场景下全量微调与 PEFT 的实用取舍参考,适合数据有限、标注稀缺的医院场景评估。

Original · arXiv cs.LG

Efficient Self-Supervised Adaptation of 3D Abdominal Vision-Language Model for Institution-Specific HCC Classification via Full Fine-Tuning and PEFT

Medical vision-language models (VLMs) capture cross-modal relationships between image and text, yet their adaptation to institution-specific clinical tasks remains underexplored. In this study, we fine-tuned a pretrained 3D medical VLM for hepatocellular carcinoma (HCC) classification using paired abdominal CT scans and radiology reports from a different institution with acquisition characteristics that differ from the model's original pretraining corpus. We compared two adaptation strategies: full fine-tuning and parameter-efficient fine-tuning (PEFT), motivated by the common use of PEFT to reduce computational cost and enable adaptation under limited-data constraints. Both approaches achieve strong downstream HCC classification performance despite the cross-institutional domain shift, with PEFT reaching an AUC of 0.79 and F1 of 0.70, and full fine-tuning achieving an AUC of 0.82 and F1 of 0.71. Although these results remain below the performance reported by supervised HCC classification approaches, our method achieves improvements over the pretrained model through CT-report contrastive learning without requiring lesion-level annotations or HCC classification labels as training targets. Full fine-tuning converges rapidly but overfits within a few epochs, whereas PEFT (ConvLoRA for the image encoder and LoRA for the text encoder) attains comparable performance while updating approximately 1% of the model parameters, although requiring more training steps. We also examine the role of contrastive temperature, observing that temperature initialization significantly affects classification performance. This study demonstrates that a 3D medical VLM can be efficiently adapted to institution-specific HCC classification using self-supervised CT-report contrastive learning, while highlighting the practical trade-offs between full fine-tuning and parameter-efficient fine-tuning.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include medical imaging, VLM, LoRA, fine-tuning.

Source attribution: curated from arXiv cs.LG (@Febryan Putra Kartika, Cheng-Yu Ma, Ying-Jia Lin, Chi-Tung Cheng, Kuan-Fu Chen).