这篇论文解决了LLM错误预测中一个被忽视的关键问题——输入模糊性会干扰UQ指标的有效性。做模型可靠性评估或安全部署的团队，建议看看他们如何用模糊性标签提升预测精度，直接可用。
该论文研究了在大型语言模型（LLM）的错误预测任务中，如何区分输入固有的模糊性（aleatoric uncertainty）与模型知识不足导致的不确定性。作者发现，传统的不确定性量化（UQ）指标在无歧义样本上预测错误更准确，而在有多个合理答案的模糊问题上表现较差。他们提出了一种方法，通过门控专家（Gated Experts）和选择性预测（Selective Prediction）将模糊性标签融入错误预测流程。实验表明，加入模糊性信息后，多个UQ指标在标准数据集上的错误预测性能提升了超过10个PRR点，且效果跨模型族、训练范式和数据集的泛化。
The Role of Ambiguity in Error Prediction via Uncertainty Quantification
The task of Error Prediction, namely predicting whether a model output is correct, is commonly tackled with Uncertainty Quantification (UQ). However, while uncertainty metrics capture when models lack knowledge or capacity to make a prediction, they also reflect aleatoric uncertainty, which is inherent in the model input and context. This paper presents a method for improving error prediction for Large Language Models (LLMs), by disentangling input ambiguity from UQ signal. We conduct experiments on the task of Question Answering (QA) with six UQ metrics and show that UQ metrics are more predictive of errors on unambiguous instances than on questions with multiple plausible answers. We use Gated Experts and Selective Prediction to incorporate gold and predicted ambiguity labels into the error prediction pipeline. We find that ambiguity information improves error prediction scores across model families, training and evaluation paradigms, datasets (including allegedly unambiguous ones), and sources of aleatoric uncertainty, yielding improvements of over 10 points of PRR for individual UQ metrics on standard datasets.