FrontierCode 把 AI 编程评测从「能跑就行」升级到「能合并才算数」，做代码质量评估或 AI 编程工具的团队值得关注——它暴露了当前模型在真实代码审查中的致命短板。
Cognition 推出 FrontierCode 编码基准测试，评估 AI 生成的代码是否达到人类维护者愿意合并的质量，而不仅仅是能否通过测试。该基准包含 150 个任务，由 20 多位开源维护者设计，每个任务耗时超 40 小时。结果显示，最强模型 Claude Opus 4.8 在最高难度 Diamond 子集上仅得 13.4%，GPT-5.5 得 6.3%，Gemini 3.1 Pro 得 4.7%。评分系统引入“阻塞项”机制，任何导致无法合并的问题（如行为错误、不安全改动）直接判 0 分，通过后才按可读性、类型安全等软质量项加权。这揭示了当前 AI 编程助手在代码设计、约束和项目风格适配上的严重不足。
Incredible! This is just the benchmark we needed. …
Incredible! This is just the benchmark we needed.
Claude Opus 4.8, achieves a score of only 13.4%. Other models score even lower: GPT-5.5 receives 6.3%, Gemini 3.1 Pro 4.7%, and others even less.
Cognition is introducing FrontierCode, a coding benchmark built to test whether AI code is good enough for a real maintainer to merge, not just whether it passes tests.
FrontierCode asks a harder question: did the model produce a clean, limited, well-tested, readable patch that fits the project’s existing style and would survive serious code review?
They bring 3 nested subsets of FrontierCode at increasing difficulty: The benchmark contains 150 tasks, with Main as the hardest 100 and Diamond as the hardest 50.
More than 20 open-source maintainers helped design the tasks, and each task took over 40 hours to build, review, attack, and calibrate.
The biggest finding is that top models still struggle badly when the target is mergeable code instead of merely working code.
On Diamond, the best model, Claude Opus 4.8, scores only 13.4%, while GPT-5.5 scores 6.3%, Gemini 3.1 Pro scores 4.7%, and the best open-source model listed, Kimi K2.6, scores 3.8%.
Shows that today’s strongest coding agents can often patch behavior, but they still fail many human-review standards around design, restraint, test quality, and project conventions.
The mechanism is a grading system built around blockers and non-blockers.
A blocker is something that would stop a maintainer from merging the PR, such as broken behavior, missing required behavior, unsafe scope changes, bad performance, or tests that do not prove the fix.
A solution that fails any blocker gets 0, even if parts of the code look good.
A passing solution then gets a weighted score based on softer quality items such as readability, type safety, style, and fit with the existing codebase.
FrontierCode also adds checks beyond normal unit tests.
Reverse-classical testing runs the model’s own tests against the original broken code, and those tests must fail, which proves the model wrote tests that actually catch the bug.
Scope checks punish patches that touch unrelated files, add oversized diffs, or refactor things the task did not ask for.
Adaptive grading uses an LLM to adjust test scaffolding around valid implementation differences, so a good solution is not rejected just because it used a different function name or error wording.