Nvidia 论文：用首个通过测试的编辑给基座模型排序，相关性达 0.964
Nvidia 发的这篇论文教你在花大钱 post-train 之前，用一个便宜方法挑出最适合做 agent 的基座模型，和 SWE-bench 排名对得上 0.964。
Nvidia 的论文指出，agent 基准测试给基座模型的分数接近零，简单编码测试也难以预测后续 agent 能力。他们提出的方法是回放一个强 agent 的成功运行，在每次编辑后重跑测试，找到首个让测试通过的编辑，然后检验基座模型是否会写出该编辑。10 个基座模型按此方法的排序与其 post-training 后的 SWE-bench Verified 分数排名一致，相关性达 0.964。该方法只需一个基准的成功 agent 运行记录和测试集，可在昂贵的 post-training 之前用来挑选基座模型。
New Nvidia paper shows scoring base models on every step of an agent run works worse than scoring them only on the edit that first makes the tests pass.
Post-training is expensive, so picking the right base model matters. Yet agent benchmarks score base models near zero, and simple coding tests barely predict later agent skill.
They replay a strong agent's successful run and rerun the tests after each edit to find the 1st edit that makes them pass. At this decisive step they test whether a base model favors that edit, picks it over failing edits, and can write a passing one.
How likely each base model was to write that exact edit ranked the same 10 models in line with their post-trained SWE-bench Verified scores, at 0.964.
When choosing a base model to post-train for agent work, run this check first. It needs only a benchmark's successful agent runs and its tests.
