RITA:测试时提示自适应提升VLM对抗鲁棒性
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
这篇论文用最优传输做分布对齐来对抗扰动,不用改模型结构,挺实用的思路。
预训练的视觉语言模型如CLIP在零样本泛化上表现优异,但在对抗扰动下性能显著下降。现有测试时自适应方法依赖样本级置信度启发式,无法区分对抗性误预测和语义一致性。
作者发现对抗扰动虽破坏整体表征,但语义在增强视图的分布中得以保留,据此提出RITA框架。RITA使用最优传输对齐增强视觉特征分布与文本原型,并引入动态缓存积累可靠线索进行在线优化。
实验表明RITA在提升对抗鲁棒性的同时,不损失干净准确率。
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This sample-centric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that RITA significantly improves adversarial robustness without compromising clean accuracy.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include CLIP, RITA, multimodal, AI safety.
Source attribution: curated from arXiv cs.LG (@Xingyu Zhu, Huanshen Wu, Shuo Wang, Beier Zhu, Jiannan Ge, Jiaheng Zhang, Long Chen).