这篇论文分析了不同评分规则对LLM预测行为的影响，帮你理解如何选择合适的训练目标。
该研究比较了五种不同的评分规则作为二元预测训练目标。Brier训练的模型在Brier分数和AUC-ROC指标上表现最佳，而log训练的模型在log分数和校准误差上最优。不同评分规则下的模型在偏差、信息和噪声组合上存在差异，即使总体性能相近。
How Proper Scoring Rules Shape LLM Forecasting
This paper evaluates how reward function choice shapes the performance and behavior of LLM forecasters. We compare five proper scoring rules as training objectives for binary forecasts of resolved real-world events. Although the rules share the same theoretical incentive for truthful probability reporting, the resulting models differ in calibration, probability use, and estimated profiles of bias, information, and noise, with smaller differences in aggregate accuracy and discrimination. The Brier-trained model has the lowest observed Brier score and highest AUC-ROC, while the log-trained model has the highest observed log score and lowest calibration error. Models with similar aggregate performance also reach that performance through different combinations of bias, information, and noise. Proper scoring rules therefore need not behave interchangeably as training objectives. Reward choice may shape not only how well an LLM forecasts, but how its forecasting errors are structured. Each condition uses a single seed, so some differences may reflect training stochasticity.