这篇论文用22种语言、9种方法实测发现:让模型用英文思考能大幅改善低资源语言的不确定性估计,选方法还得看模型大小。
该研究首次在22种语言(涵盖高、中、低资源)上系统评估了9种不确定性估计方法。发现用英文推理(即使问题为低资源语言)能显著提升UE性能,并缩小与高资源语言的性能差距。研究还指出,小规模模型下基于概率的开放盒方法更优,大规模模型下封闭盒的自我表述不确定性更有效。论文提供了多语言选择性预测中阈值选择的指导。
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22 languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q\&A datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form reasoning, avoiding LLM-as-a-judge and embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting that comprehension of low-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource languages, demonstrating that generation language matters more than the question language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings.