Singularity-Bench 发布：测试智能体能否端到端构建 AI 系统
Singularity-Bench: Can Agents Build Agents?
这个基准让智能体从零写训练、推理、编排系统，再和人类基线硬碰硬。结果挺扎眼：Opus 5.5 配 Claude Code 六项全过人类线，其他智能体平均只有 0.58。做 agent 评测或基础设施的工程师都该看看。
研究人员推出 Singularity-Bench，一个开放式基准，涵盖预训练、中期训练、后训练、推理、agent harness 和集群编排六项端到端任务，要求智能体从最小起点实现核心 AI 系统并与人类工程基线在同硬件下对比。
基准还通过覆盖 OWASP Agentic Top 10 中 9 类攻击的红队测试，暴露了 Harbor 框架的奖励劫持漏洞，并推出能阻断全部测试攻击的扩展 Breakwater。在 11 个由 7 个前沿模型和 2 个 harness 组成的智能体中，Opus 5.
5 配合 Claude Code 在全部六项任务上达到或超过人类基线，Singularity-Index 达 1.25，其余智能体平均仅 0.
58。团队希望该基准追踪智能体在 AI 开发中的自动化程度，以及何时能从辅助转向独立构建更强的后继系统。
Singularity-Bench: Can Agents Build Agents?
Frontier AI agents increasingly succeed on realistic, long-horizon tasks and are beginning to improve the systems through which they operate. Yet existing benchmarks often restrict agents to narrow action spaces, use simplified environments, or rely on verification methods that are unreliable for evaluating arbitrary agent-generated systems. We introduce Singularity-Bench, an open-ended benchmark for autonomous AI development with six end-to-end tasks spanning pre-training, mid-training, post-training, inference, the agent harness, and cluster orchestration. Each task requires agents to implement core systems from a minimal starting point and evaluates them against mature human-engineered baselines on identical hardware. Additionally, through extensive red-teaming covering 9 of the 10 categories in the OWASP Top 10 for Agentic Applications, we expose reward-hacking vulnerabilities in the conventional Harbor framework and develop Breakwater, an extension of Harbor that blocks all attacks tested in our study. Across 11 agents built from 7 frontier models and 2 harnesses, Opus 5.5 with Claude Code matches or outperforms the human baseline on all six tasks with a Singularity-Index of 1.25, suggesting it may have the skills needed to reach the singularity. The remaining agents average only 0.58. Our vision is for Singularity-Bench to track how much of AI development agents can already automate and when they may transition from assisting to taking end-to-end responsibility for building stronger successors.
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Claude Code, Singularity-Bench, benchmark, 智能体.
Source attribution: curated from arXiv cs.AI (@Yang Zhou, Krish Agarwal, Zhuoming Chen, Zerui Wang, Zhenyu Zhang, Wenshuang Li, Xinpeng Wei, Zhichen Zeng, Jingyu Liu, Jue Wang, Sen Wu, Ziheng Jiang, Zhen Dong, Han Cai, Binhang Yuan, Ying Sheng, Ce Zhang, Bo Li, Beidi Chen).
