New method calibrates test-time prompt tuning
Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuning
This research provides a practical solution for improving the reliability of AI models during inference, which is crucial for applications where model confidence must be trustworthy.
A new method called CoTS applies temperature scaling to calibrate test-time prompt tuning (TPT), which adapts models on a single test instance.
CoTS preserves accuracy while reducing miscalibration.Experiments show E-CoTS, an ensemble version of CoTS, cuts expected calibration error on ImageNet variants from 11.
90% to 5.38% and boosts accuracy from 60.
74% to 62.95%.
Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuning
Test-time prompt tuning (TPT) enables adaptation on a single test instance, achieving improved accuracy but often sacrificing calibration performance. Most existing calibration methods introduce additional regularization terms to promote dispersion across text embeddings and reduce calibration error, yet these methods often suffer from a drop in accuracy. Motivated by the well-calibrated nature of zero-shot predictions, we propose CoTS, a simple yet effective post-hoc calibration method that preserves accuracy. Specifically, CoTS applies temperature scaling to minimize the confidence gap between adapted and zero-shot predictions. To fully exploit the potential of multiple augmentations during adaptation, we introduce a weak-strong ensemble strategy that further boosts accuracy. We then apply CoTS to this ensemble, termed E-CoTS, to maintain its well-calibrated property. Extensive experiments on diverse datasets and backbones show that our approaches effectively mitigate miscalibration without compromising primary accuracy. For instance, E-CoTS reduces the average expected calibration error of TPT from 11.90% to 5.38% on ImageNet variants, while even increasing accuracy from 60.74% to 62.95%. Moreover, when integrated with existing calibration methods, E-CoTS usually enhances both accuracy and calibration simultaneously.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Calibrating Test-Time Prompt Tuning, prompt engineering, calibration.
Source attribution: curated from arXiv cs.LG (@Yuwei Liang, Jian Liang, Dapeng Hu, Yinuo Xu, Ran He).