他们揪出了SciCode基准里的263个坑，修完后模型分数从60%直接飙到98%——原来不是模型笨，是考卷错了。
SciCode是评估语言模型科学编码能力的标准基准，但近期模型分数停滞在60%子问题准确率附近。研究者审计全部65个问题，发现263处缺陷，其中192处导致正确的指令遵循代码被误判。修正后推出SciCode-Verified，重新测试12个前沿模型，子问题准确率从45-60%升至84-98%，主问题准确率从9-27%升至69-92%。结果表明瓶颈在于基准质量而非模型能力。
SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models
SciCode is the standard measure of the scientific-coding ability of language models: research-level problems that demand both frontier scientific theory and its implementation as working numerical code. It is a component of the Artificial Analysis Intelligence Index and a standing evaluation in government and national-laboratory suites. Yet its scores have recently plateaued: the strongest 2026 models cluster tightly around 60\% subproblem accuracy, and a successor model ties its predecessor. We trace this stagnation to defects in the benchmark itself. A per-problem, domain-expert audit of all 65 test problems uncovers 263 defects; 192 of them, spread across 91\% of the main problems, cause correct, instruction-following solutions to be wrongly rejected---through non-reproducible gold answers, over-tight tolerances, or self-contradictory specifications. Critically, 78\% of these score-suppressing defects require specialized physics or mathematics knowledge to detect, not mere clerical proofreading. We corrected every confirmable defect to produce SciCode-Verified. The corrections add only the specifications a well-posed problem requires, repair grading, and tighten the tests that were too lenient; every change is recorded with its justification and independently re-checked by a second domain expert. We re-evaluate twelve frontier model snapshots on the corrected benchmark and find a substantial recovery: subproblem accuracy rises from 45--60\% to 84--98\%, and main-problem accuracy from 9--27\% to 69--92\%. State-of-the-art models are far more proficient in scientific coding than SciCode has suggested---the bottleneck was not model capability, but the quality of the evaluation instrument. We release SciCode-Verified with its complete audit trail as the corrected public standard.