SPLIT 方法用小样本试验分配测试时算力，误差不超过 0.8 个百分点
Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers
测试时多采样加验证器很常用，但预算怎么分全靠拍脑袋。这篇论文给出 SPLIT，用小试点数据算出接近最优的分配方案，还公开了 26 个模型上的实测结果。
论文提出 SPLIT，一种在推理阶段分配算力的方法：通过小规模标注试点估计一个精确公式，来决定生成多少候选答案、用哪些模型生成、花多少预算做验证。该方法考虑了任务难度差异、模型共有的盲区和验证器重复调用之间的相关性错误。在覆盖数学、科学问答、代码和软件工程智能体的 7 个基准、26 个模型、175 个智能体系统和 6 种真实验证器上，SPLIT 与事后最优分配的差距在 0.02-0.8 个百分点内，而测试的所有固定规则在部分场景至少损失 2.5 个百分点。公式还解释了增加 LLM 评审数量何时反而降低精度。
Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers
A common way to make language models and agents more accurate is to spend more compute at inference time: generate many candidate solutions and submit the one a verifier prefers. Success then requires that some candidate is correct and that the verifier picks it. The first grows predictably with the number of samples; the second does not, because real verifiers (unit tests, majority votes, LLM judges, reward models) accept wrong answers, reject right ones, and make correlated mistakes. With a fixed budget, a practitioner must decide how many samples to draw, which models to draw them from, and how much to spend on verification, and today these choices are made by rules of thumb. We introduce SPLIT, which makes them from a small labelled pilot. It rests on an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection that accounts for varying task difficulty, blind spots shared across models, and correlated repeated verifier calls. SPLIT fits the formula by empirical Bayes and uses it to plan budgets on unseen tasks. Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting. Regret depends only on how well SPLIT ranks configurations, and it stays near-optimal under compute, price and latency costs and online. The formula also explains when asking more LLM judges lowers precision. As a forecaster far beyond the pilot, SPLIT beats power-law fitting when the verifier caps success, and we report where it loses.
