无信号选择与表达恢复:冻结小代码模型后验验证算子测量研究

Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models

精选理由

这篇论文测了26种后验操作,结论是除了M1都没跑过最朴素的BoN。M1靠恢复被遗漏的正确代码,让DeepSeek-Coder-1.3B在HumanEval+上多解对12题,值得搞代码生成的同学看看。

AI 摘要

针对参数不超过1.5B的冻结小代码模型,研究在无需微调下通过后验验证操作提升程序正确性的效果。在匹配计算量的泄密自由协议下,评估了26种语义后验算子(选择、验证、修复、消除等),均未能在HumanEval+和MBPP+上超越Best-of-N (BoN)的留出准确率。研究发现覆盖墙、能力剪刀差和空共识陷阱是根本原因。唯一带来精度提升的算子M1通过表达层恢复标准提取器丢弃的正确程序,使DeepSeek-Coder-1.3B在HumanEval+上多解+12题(p=2.4e-4),且零伤害。自适应共识早停(ACE)节省约19%计算量且无伤害。

原文 · arXiv: DeepSeek

Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models

Frozen small code models (<=1.5B parameters, run locally without fine-tuning) suit offline and privacy-constrained use, but often emit plausible-but-wrong programs. A natural remedy is a post-hoc operator that selects, verifies, repairs, or re-processes the model's samples without retraining; in principled form it is Popperian: attack each candidate with a severe test, keep what survives. We measure whether such operators help. Under one deterministic execution oracle and a leakage-free, matched-compute protocol, 26 semantic post-hoc operators (selection, verification, repair, elimination, portfolios, sound vetoes, generation conditioning) are evaluated against Best-of-N (BoN); on the cells and benchmarks tested, none improves held-out accuracy over BoN. The negative is mechanistic: a coverage wall (systematic hard-task failures deeper sampling does not rescue), a capability scissors (a competent generator leaves almost no discriminable error among visible-test passers), and a near-empty consensus trap (the visible-pass-but-hidden-wrong majority a leakage-free selector needs rarely co-occurs with a correct alternative). A distribution-free do-no-harm bound cannot certify a harm rate <=alpha at zero observed harm unless n>=45. Two operators help on a different axis, outside the semantic output space. An expression-layer recovery (M1), the only accuracy gain here, recovers correct programs the standard extractor discards (robust extraction and public-test signature alignment); it does no harm (b10=0), is leakage-free, and lifts DeepSeek-Coder-1.3B by +12 tasks on HumanEval+ (p=2.4e-4). An adaptive consensus early-stop (ACE) is a calibrated compute-saving control (~19% saving, zero harm). M1 and the selection negative replicate on HumanEval+ and MBPP+ across three model cells. The lesson: fix the harness and measure coverage before blaming semantic post-hoc reasoning.