新方法：用训练后轨迹预测基座模型的编程智能体潜力
Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models
训练一个编程智能体很烧钱，这篇论文教你怎么先挑基座模型，用 SWE-bench 的现成轨迹就能筛，方法挺巧的。
一篇 arXiv 论文提出在昂贵的 agentic post-training 之前预测基座模型潜力。作者将训练后智能体的成功轨迹作为前瞻信号，通过重放轨迹并逐步重跑测试，定位使仓库从失败转为通过的决定性步骤。在此基础上构建三种筛选指标：Decisive-Action BPB、Patch MCQ 和 prefix-conditioned pass@K。在十对公开基座与训练后模型上，三种指标与 SWE-bench Verified pass@1 的排序高度一致。
Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models
How can we predict which base checkpoint is worth an expensive round of agentic post-training? End-to-end pass@$K$ tests whether successful behavior already appears in a base model's distribution, but it is a poor fit for agentic coding: many base checkpoints cannot reliably produce the well-formed tool invocation required to complete a task end-to-end. Single-shot or short-horizon tasks avoid these tool-calling failures by collapsing a multi-step interaction into a fixed prompt and a single patch, but they sidestep the core capability we care about: maintaining coherent state over many tool-using steps as the repository evolves. To bridge this gap, we treat successful post-trained agent trajectories as a lookahead signal of base-model potential. Replaying each trajectory and rerunning tests after every code-changing step identifies the decisive step: the first step whose cumulative patch flips the repository from failing to passing, certifying that the recorded action solves the task given the prior context. Motivated by a coverage principle for agentic traces, we build three screens at this step that do not require a base checkpoint to drive the harness from a cold start: (i) Decisive-Action BPB (bits per byte) measures the probability mass on the certified action, (ii) Patch MCQ tests the checkpoint's choice between that action and alternatives rejected by the same verifier, and (iii) prefix-conditioned pass@$K$ evaluates support for functionally-correct generations and credits any continuation that the tests accept. Across ten pairs of public base and post-trained models, all three screens rank the cohort in close agreement with post-trained SWE-bench Verified pass@$1$. As our methods need only a benchmark's successful trajectories and its verifier, they can be applied to turn future agentic coding benchmarks into base-model evaluations.
