做AI智能体评估或排行榜的团队终于有了统计严谨的不确定性量化工具——无需分布假设即可保证覆盖，还能处理多智能体管道和排名稳定性问题，建议做评估基准的开发者直接看论文和代码。
该研究将分裂共形预测和自适应共形推断（ACI）应用于连续AI智能体评估，提供无分布假设的覆盖保证。在24小时预测窗口内，共形区间在所有名义水平上的校准误差低于0.02，ACI在智能体发布后正确将区间扩大35%后重新收敛。研究还开发了多智能体管道的组合不确定性界限、成对排名的共形弃权规则（控制假排名率）以及排行榜级多重检验的FDR校正弃权。通过每小时收集18个实时信号评估50个智能体，发现每个智能体的条件覆盖集中在名义水平附近（均值80.4%，90%的智能体在[72%,90%]内），跨来源情感分歧可预测排名不稳定性（r=0.64, p<0.01）。代码和数据已以CC BY 4.0协议发布。
Distribution-Free Uncertainty Quantification for Continuous AI Agent Evaluation
We adapt split conformal prediction and adaptive conformal inference (ACI) to continuous AI agent evaluation, providing distribution-free coverage guarantees for forecasted quality scores. Conformal intervals achieve calibration error below 0.02 across all nominal levels at the 24h horizon, while ACI correctly widens intervals by 35% following agent releases then reconverges. We further develop compositional uncertainty bounds for multi-agent pipelines (validated via simulation across inter-stage correlations rho in [-0.5, 0.9]), a conformal abstention rule for pairwise rankings with controlled false-ranking rate, and FDR-corrected abstention for leaderboard-scale multiple testing. Evaluating 50 agents via 18 real-time signals collected hourly, we show that per-agent conditional coverage is well-concentrated around the nominal level (mean 80.4%, 90% of agents within [72%, 90%]), and that cross-source sentiment divergence predicts ranking instability (r=0.64, p<0.01). A circularity-controlled validation confirms the framework captures signal beyond benchmarks (rho_s=0.52, p<0.01, n=35). Code and data are released under CC BY 4.0.