CMU 与 JetBrains 论文：token 消耗不等于工程师产出
公司现在流行拿 token 消耗给工程师排名，这篇 CMU 和 JetBrains 的论文用 404 名工程师的数据说明这么测不准：花得多人产出未必多。管工程团队、定 AI 预算的人都该看看怎么改考核方式。
Carnegie Mellon 与 JetBrains 的研究将 5 周编程 Agent 的 token 消耗数据与 404 名 JetBrains 工程师的 commit 记录关联分析，并对 412 名员工做了问卷调查，技术人员在近九成评估中报告将 Agent 用于编程之外的工作。研究发现，同一名工程师 token 消耗翻倍，commit 数量仅增加约 10%。
更值得注意的是，在 commit 数量相近的周之间，第 90 百分位的周 token 消耗是第 10 百分位周的 39 到 240 倍，说明 token 用量更多反映工程师使用 Agent 的频率和习惯，而非实际交付成果。论文指出，一些公司已开始用 token 消耗制定预算甚至给员工排名，这种做法可能误判生产力。论文建议 token 数据只用于预算管理，评估 Agent 辅助工作应以任务和结果证据为准。
New Carnegie Mellon and JetBrains paper finds that token spend shows how much engineers use AI agents, not how much they ship.
Companies now track engineers' token spend to set budgets and judge productivity, and some even rank top users. No study had followed both spend and output for the same engineers over time.
They linked 5 weeks of coding-agent token use to the commits of 404 JetBrains engineers. They also analyzed a company survey of 412 employees, where technical staff reported using agents beyond coding in nearly 9 of 10 evaluations.
For the same engineer, doubling spend went with only about 10% more commits. Among weeks with similar commit counts, the 90th-percentile week used 39 to 240 times the tokens of the 10th-percentile week.
Use token spend only for budgeting, and judge agent-assisted work by evidence about tasks and outcomes.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include JetBrains, 编程助手, token 消耗, 工程效能.
Source attribution: curated from rohanpaul_ai (@rohanpaul_ai).
