Meta 提出 Benchmark Factory:从内部代码变更自动生成编码评测基准
很多公司想用自己代码库测模型,但做基准又贵又难。Meta 这套 Benchmark Factory 让智能体自动从内部代码变更里生成评测,还带质量检查,论文里有验证数据,工程团队值得照着试试。
Meta 发表论文提出 Benchmark Factory,一种用智能体循环从团队已合并的内部代码变更中自动生成编码评测基准的方法。它通过编写查询来选择任务,用已知相对强度的模型进行校准,并自动修复基准直到通过一组质量检查。质量检查无需逐任务的人工标注。为验证检查的有效性,作者故意构造了劣质基准:过简单的基准被拒绝,包含五种不同类型评分错误的基准也被识别。将模型身份打乱后,劣质基准漏检率最高仅 1.4%。在未参与构建的配置上,这些基准将 Claude Opus 4.7 高 effort 档排在高 effort 60.6% 对低 effort 54.4%,GPT-5.5 以 65.8% 对 60.5% 超过 GPT-5.4,10 组团队级别对比中有 9 组排序一致。论文地址为 arxiv.org/abs/2611.18890。
技术上,Benchmark Factory 把基准构建变成一个可自我验证的流程:智能体根据代码库的真实变更生成任务与查询,再用模型强度的已知差异作为锚点检验基准是否能正确区分强弱模型,不达标就继续修复。这一设计解决了自定义编码基准难以制作的核心痛点,企业可以用自己的私有代码库评测模型,而不必依赖通用基准。
论文地址:arxiv.org/abs/2611.18890,相关解读见 academy.dair.ai。
Interesting paper from Meta showing how to build coding evals from your own codebase. (bookmark it) Good benchmarks are hard to create, but using this method with agents looks promising. They propose Benchmark Factory, an agent loop that turns merged internal code changes into benchmarks. It writes the query that selects tasks, runs models of known relative strength, and repairs the benchmark until it passes a set of quality checks. These checks need no per-task human labels. To make sure the checks catch bad benchmarks, the authors fed them bad benchmarks on purpose. A benchmark that was far too easy got rejected. So did benchmarks with five different kinds of broken grading. When they scrambled which model was which, a bad benchmark slipped through at most 1.4% of the time. On configurations never used to build them, the benchmarks ranked Claude Opus 4.7 at high effort above low effort (60.6% vs 54.4%) and GPT-5.5 above GPT-5.4 (65.8% vs 60.5%). 9 of 10 per-team comparisons agreed. arxiv.org/abs/2611.18890 Chat with Paper: academy.dair.ai/papers/benchma…📊 3 ⚡
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Benchmark Factory, Meta, Claude Opus 4.7, GPT-5.5.
Source attribution: curated from elvis (@omarsar0).