这篇论文用最优传输做分布对齐来对抗扰动，不用改模型结构，挺实用的思路。
预训练的视觉语言模型如CLIP在零样本泛化上表现优异，但在对抗扰动下性能显著下降。现有测试时自适应方法依赖样本级置信度启发式，无法区分对抗性误预测和语义一致性。作者发现对抗扰动虽破坏整体表征，但语义在增强视图的分布中得以保留，据此提出RITA框架。RITA使用最优传输对齐增强视觉特征分布与文本原型，并引入动态缓存积累可靠线索进行在线优化。实验表明RITA在提升对抗鲁棒性的同时，不损失干净准确率。
Robustifying Vision-Language Models via Test-Time Prompt Adaptation
Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This sample-centric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose RITA, a Robust test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that RITA significantly improves adversarial robustness without compromising clean accuracy.