做LLM评估或模型对齐的团队会关心：这篇论文给出了在有限标注预算下选择校准策略的实用指南，建议直接参考其机制图来优化你的法官面板配置。
这篇论文研究了在有限人工标注预算下，LLM法官面板（多个LLM作为评估者）的校准策略选择问题。低维堆叠器（如标量或可靠性聚合）估计成本低但无法捕捉交互效应，而联合输出表可以建模交互但需要更多数据填充单元格。作者提出了一个有限校准机制图，并实例化为可部署的验证选择器。在RewardBench、LLMBar等基准测试中，使用7个法官（包括DeepSeek V4 Flash）的实验表明，标量/可靠性聚合在20个真实数据集-预算组合中赢了16个，说明当前法官输出往往是加性或冗余的。当存在六路交互时，联合输出表才显著优于标量方法（测试MSE从0.224降至0.061）。结论是，关键问题不是“需要多少法官”，而是下一个法官的信息在当前人工标签下是否可估计。
A Finite-Calibration Regime Map for LLM Judge Panels
We study when LLM judge panels should be calibrated with low-dimensional stackers versus joint output tables under finite human-label budgets. Low-dimensional stackers have small estimation cost but miss interactions, whereas joint-table calibrators can represent interactions but pay for cell counts and unseen patterns. We cast this tradeoff as a finite-calibration regime map and instantiate it as Finite-Calibration Panel Selection, a deployable validation selector over judge path, prefix size, and aggregator family with table and parametric estimation diagnostics. On RewardBench, LLMBar, SummEval, and Arena100K with a seven-judge pool including DeepSeek V4 Flash, scalar/reliability aggregation wins 16 of 20 real dataset--budget cells, indicating that current judge outputs are often additive or redundant. Controlled calibration-growth data show the complementary regime: additive labels remain scalar-favored, whereas a six-way interaction selects a larger joint table and its test MSE drops from 0.224 to 0.061 once unseen mass vanishes. Thus the practical question is not ``how many judges?'' but whether the next judge's information is estimable under the available human labels.