ExpertCrowd-SWE发布：100个专家工程任务测试编程智能体
ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases
想让编程智能体在陌生大仓库里干真活的人该看看这个基准。它不是改几行的小修小补，而是工程师写的真实变更请求，评分器能识别偷懒和作弊补丁。榜单上 Claude Opus 5.5 和 GPT-6 Astra 基本打平，选型做对比时很实用。
ExpertCrowd-SWE（EC-SWE）是一个新的软件工程基准，包含100个由在职工程师撰写的多文件工程任务，覆盖87个开源项目和11种编程语言。每个任务针对真实仓库的固定 commit 提出变更请求，中位数工单长148词，参考解决方案中位数改动471行。
评测机制上，每个任务配有密封参考答案和任务专属评分器，采用加权、可执行的标准给出部分分；必须通过的阻断项模拟维护者拒绝合并的底线。近半任务附带惰性补丁和对抗性补丁，用于检验评分器能否正确区分。所有任务图像在发布前校准，确保参考答案得1.0分、未改动仓库和无效改动得0分。
评测涵盖七个前沿模型、每个模型五轮运行（共3495次评分运行），使用统一平台为各模型提供原生工具接口。Claude Opus 5.5以67.3%领先，GPT-6 Astra以63.7%紧随，两者差距无统计显著性；Claude Fable 5.1得46.5%，Gemini 3.8 Flash得40.0%，Qwen 3.8 Max、DeepSeek V4.1 Flash和Grok 4.6处于32-34%区间。没有任务被所有模型完全解决，80个任务上没有任何模型在全部五轮中获得满分。团队公开24个任务的环境与结果，保留76个任务以防污染。
ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases
ExpertCrowd-SWE (EC-SWE) is a benchmark of 100 expert-authored, multi-file software engineering tasks for evaluating autonomous coding agents in large, unfamiliar codebases. Each task is a change request written by a practising engineer against a pinned commit of a real repository, paired with a sealed reference solution and a task-specific grader whose weighted, mostly executable criteria award partial credit; must-pass blockers and pass/fail gates encode what a maintainer would refuse to merge. The tasks span 87 open-source projects and 11 languages. The median ticket is 148 words, while the median reference solution changes 471 lines. Every task image is calibrated before release so that the reference solution scores 1.0 and both the untouched repository and an inert change score 0; nearly half of the tasks also ship alternative, lazy and adversarial control patches that the grader must classify correctly. The final 100 tasks were selected for difficulty against current frontier models. We evaluate seven frontier models, five runs each (3,495 scored runs), in a single platform harness that gives each model its provider-native tool interface. Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) lead without a statistically significant gap, followed by Claude Fable 5.1 (46.5%), Gemini 3.8 Flash (40.0%), and a statistically indistinguishable group of Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 (32-34%). No task is fully solved by every model, and on 80 tasks no model earns full credit on all five runs. Scores span 35 points. Depending on the model, agents take a median of 33 to 166 steps per run, with little relation to score. We release 24 tasks with their environments, graders and results, and hold out 76 to limit contamination.
This report is categorized under AITOP's "Models" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include ExpertCrowd-SWE, Claude, GPT, DeepSeek.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority. Cross-source corroboration: 11 independent sources have covered this development (including IT之家, 宝玉, berryxia), confirming industry-wide signal strength.
