SWE-CoWork benchmark tests coding agents on workplace collaboration
The 63% check pass rate versus under 8% full task completion shows agents break down not at coding but at gathering unstated requirements — a failure mode most leaderboards hide entirely. Teams evaluating agents for internal developer tools should treat requirement-seeking behaviour as a selection criterion alongside SWE-Bench style scores. Anyone building agent orchestration layers can use the per-check requirement mapping to diagnose whether their agent asks questions early or silently ships the wrong thing.
TokenWave AI has released SWE-CoWork, a benchmark that evaluates AI coding agents inside a simulated workplace instead of handing them fully specified tasks. In its first published result, GPT-6 Astra gained 16 points on passed checks when given the full specification up front, yet still fully completed only 8 of 101 tasks. The best models pass roughly 63% of individual checks but fully finish fewer than 8% of the 101 tasks.
Unlike conventional coding benchmarks that deliver a complete written brief, SWE-CoWork distributes 82% of the requirements across simulated teammates. The agent must therefore figure out what to build by asking the right person the right questions, mimicking how a new hire learns scope during a first week on the job. Each failed check is linked back to a specific requirement, making it possible to see exactly where the agent misunderstood or failed to gather context.
The benchmark exposes a gap between solving well-defined problems and operating in ambiguous, collaborative settings. As coding agents move from toys into daily engineering workflows, the ability to proactively seek requirements from colleagues may matter more than raw code generation accuracy, and SWE-CoWork gives teams a measurable way to track that capability.
Coding-agent benchmarks are starting to look like a new hire’s first week.
TokenWave AI has released SWE-CoWork, a new benchmark that tests whether AI coding agents can work like real team members.
Giving GPT-6 Astra the full spec up front lifted its passed checks by 16 points, yet it still fully finished only 8 of 101 tasks. SWE-CoWork puts coding agents in a simulated workplace modeled on real-world workflows, then links each failed check to a specific requirement.
Most coding benchmarks hand an AI agent a complete written task, but SWE-CoWork puts it in a simulated workplace where 82% of the requirements sit with teammates, so the agent only learns what to build by asking the right person. The best models pass about 63% of the individual checks yet fully finish fewer than 8% of 101 tasks.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include SWE-CoWork, GPT-6, coding agent, agents.
Source attribution: curated from rohanpaul_ai (@rohanpaul_ai).