这篇论文发现直接用余弦相似度搞局部共形预测效果一般，但换个非线性变换后，预测集变小了，值得做VLM不确定性量化的看看。
本文对视觉-语言模型（VLMs）在自然图像分类任务上使用局部化共形预测（localized conformal prediction）进行不确定性量化。作者通过开源实现基准测试，发现直接使用测试样本与校准样本视觉特征的余弦相似度并不能优于非局部基线。他们提出一种简单的非线性变换，在保持边缘覆盖保证的同时，实现了统计显著的集合大小平均减少。实验在多个VLM上验证了有效性。
Localized Conformal Prediction for Image Classification with Vision-Language Models
Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees. Full conditional guarantee is not an attainable goal, a well known fact in conformal predictions literature. As a result, several approaches have tried to approximate this behavior by adapting the conformal sets of test-time samples according to their similarity to calibration examples. Although the latter has gained traction and shown impressive performances for regression problems, its application to image classification remains under-explored. We conduct an extensive benchmarking on natural image classification tasks with vision-language models (VLMs), using our open source implementation of a recent localized conformal prediction algorithm. We show that straightforward usage of the cosine similarity between test-time and calibration visual features, an intuitive choice for VLMs, is not sufficient to improve over the non-local baselines. In response, we propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction. Code is available at https://github.com/cfuchs2023/lcp-vlm/.