GitHub Copilot agentic harness 基准测试:性能持平但 token 更省
GitHub 实测了 Copilot 智能体框架,五个基准上不输原生,还省 token,支持 20 多种模型,值得试试。
GitHub 在 SWE-bench Verified、SWE-bench Pro、SkillsBench、TerminalBench、Win-Hill 五个基准上对 Copilot agentic harness 进行了对比测试。固定模型和任务后,任务解决率与模型原生 harness 持平。在大多数配置下 token 消耗更少,最高可省 30%。Copilot 支持超过 20 个模型,用户可针对任务自由切换效率或质量。
We benchmarked the GitHub Copilot agentic harness against the harnesses that ship leading models natively. Holding the model and task fixed across SWE-bench Verified, SWE-bench Pro, SkillsBench, TerminalBench, and Win-Hill, the results were clear: ✅ Task resolution on par with model-vendor harnesses ✅ Fewer tokens across most configurations 💡 A key learning: With GitHub Copilot supporting more than 20 models, you're free to pick efficiency or peak quality per task. 💬 2 🔄 1 ❤️ 19 👀 5894 📊 4 ⚡