做 SWE 智能体评估或开发 AI 编程助手的团队，这个新基准更贴近真实开发场景，值得关注其设计思路和模型表现差异。
DeepSWE 是一个新的软件工程/智能体基准测试，包含 113 个任务，覆盖 91 个仓库和 5 种编程语言。其评估框架 mini-swe-agent 为每个模型提供单一的 bash 工具和相同的系统指令，没有厂商自定义原语。评估提示比 SWE-Bench Pro 更短，但平均需要修改 5.5 倍以上的代码和 7 个文件，旨在模拟开发者与智能体对话的真实方式。初步结果显示，Claude Opus 比 Claude Code 高 10 个百分点，Gemini 3.1 Pro 比 Gemini CLI 高 20 个百分点。该基准强调指令遵循能力，可能对探索型模型不利。
Interesting new SWE/agentic benchmark (DeepSWE) was released yesterday. 113 tasks across 91 repos in...
Interesting new SWE/agentic benchmark (DeepSWE) was released yesterday. 113 tasks across 91 repos in 5 languages. Here are interesting things I noticed: - The evaluation harness (mini-swe-agent) gives every model a single bash tool and the same SI. No vendor editing primitives. - Eval Prompts are shorter than SWE-Bench Pro, but require 5.5× more code and touch 7 files on average. The idea is to mimic how developers actually talk to agents, short behavioral descriptions, not verbose specs. - SI describes a specific workflow: find code, reproduce, fix, verify, edge cases, submit. This maps directly onto how the verifier grades, which could bias toward models that follow instructions literally over models that explore more. - The bash tool is guarded, outputs over 10k chars get truncated. Malformed tool calls get caught and retried with guidance rather than crashing. To prevent to blow up context. - Mini-swe-agent claims to match or beat 1P harnesses on the same tasks. Claude Opus scored +10pp over Claude Code. Gemini 3.1 Pro scored +20pp over Gemini CLI. Would love to see how other harness × model combinations will do, e.g. @cursor_ai , @antigravity , @FactoryAI and how well the eval harness does on more general knowledge work, e.g. GDPval. Great to see the SWE-agent team keep pushing on both the research and eval side. 🤗 💬 2 🔄 1 ❤️ 6 👀 322 📊 3 ⚡
- rohanpaul_ai05-27 21:21原文