OptimismBench：语言模型概率判断的乐观偏差与对齐效应
OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment
这篇论文发现大多数大模型判断概率时普遍乐观，只有Anthropic的模型偏悲观，还揭示对齐训练会改变偏差方向，值得AI开发者和伦理研究者关注。
为了检测语言模型在概率判断中的方向性偏差，研究者提出OptimismBench，使用反向配对法计算符号偏差分数。
对8个提供商的16个模型测试发现，14个模型呈现乐观偏差，仅Anthropic的旗舰层表现悲观。11组基座模型与聊天模型对比显示，后训练方向在不同模型家族中产生相反偏差。
在17个模型、6种语言的比较中，模型身份主导语言因素，模型间方差是语言间方差的4.7倍。
研究者发布了3870个条目、覆盖10种语言的偏差审计数据集。
OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment
Large language models are increasingly used as decision aids whose probability judgments shape downstream choices. Whether those judgments carry a systematic directional tilt has been hard to detect: calibration metrics aggregate unsigned errors, and naturalistic uncertainty offers no ground-truth probability. When an LLM rates a startup's success at 70% but its failure at 15%, the missing 15 points expose a distortion no aggregate score flags. We introduce OptimismBench, which detects directional bias with inverted pairs: each scenario elicits both P(success) and P(failure), and asymmetry between the two framings yields a signed bias score without ground truth. Across 16 models from 8 providers, fourteen are optimistic; pessimism appears only in Anthropic's frontier tier. Eleven matched base-versus-chat pairs across four families show post-training sets the sign of the bias, with opposite shifts in different families. The pattern survives prompt, temperature, perspective, and self-debiasing ablations. A seventeen-model six-language comparison further shows model identity dominates language, with inter-model variance at 4.7x inter-language variance. We release 3,870 items across 10 languages for per-model directional-bias auditing. When alignment makes a model more helpful, it also tilts its probabilities; downstream pipelines inherit the tilt by default.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include OptimismBench, Anthropic, 语言模型, 偏差检测.
Primary source verification: published directly by arXiv: Anthropic with first-party authority. Cross-source corroboration: 4 independent sources have covered this development (including AI Will, Anthropic, Yangyi), confirming industry-wide signal strength.