这篇论文用实验数据告诉你,让AI反复修改代码不靠谱,并给出了带状态绑定的证据机制,做代码修复的开发者值得看看。
该研究基于30个HumanEval修复任务的900条三轮修订轨迹发现,强制修订后当前正确率从第一轮的0.820下降至第二轮后的0.673,但最终正确率升至0.847。在140亿参数模型复制实验中,使用陈旧轨迹导致34/135初始正确修复受损,而使用当前轨迹仅4/135受损(提升22.2个百分点,95% CI [8.9,37.0],Holm p=0.0337)。前瞻性540次 rollout 政策消除了初始正确的损害,但降低了错误修复能力。论文提出状态绑定证据与类型化修订契约,并在参考实现中强制可机械验证的子集。
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
Generate--test--revise loops are common in coding agents, but repetition alone provides no reliability guarantee. We study the gap between finding a correct patch and retaining, verifying, and submitting it. A sealed five-seed study over 30 HumanEval repairs produces 900 three-revision trajectories. Under forced revision, current correctness with current traces falls from 0.820 after one revision to 0.673 after two, although ever-correct rises to 0.847. Two common-state studies use 2,430 branches from identical frozen programs to remove post-treatment risk-set bias. In a prespecified 14B replication, stale traces harm 34/135 correct starts versus 4/135 with current traces, a 22.2-point increase (task-cluster 95\% CI $[8.9,37.0]$, exact Holm $p=0.0337$). A prospective 540-rollout policy eliminates observed correct-start harm but reduces wrong-start repair and fails its joint criterion. Repository experiments over 24 bugs and four coder stacks expose floor effects and component heterogeneity without Holm-significant effects. We therefore separate admission, preservation, grounded certification, competence, and liveness. We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation that binds verifier evidence to exact code states, preserves verified checkpoints, and emits auditable admission receipts. The implementation is an executable specification and conformance artifact, not evidence of improved repair competence or calibrated verifier dependence.