JetBrains 研究：工程师 token 消耗能否衡量产出？
The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures
如果你正用 token 消耗考核团队 AI 使用率，这篇 JetBrains 的研究要先看看：404 名工程师的数据显示消耗翻倍只对应约 10% 的 commit 增长。它给出了替代方案——用任务和结果评估，而不是用量。
JetBrains 团队在 arXiv 发布一项案例研究，追踪 404 名工程师五周内使用编程智能体（coding agent）的 token 消耗，并将其与 commit 记录关联，同时分析了 412 名员工的内部调查。
数据显示，token 消耗的差异主要来自工程师个体之间：在 commit 数相近的周内，消耗量第 90 百分位是第 10 百分位的 39 到 240 倍，而消耗最多的工程师名单在五周内基本保持稳定。同一工程师内部，消耗翻倍大约只对应 10% 的 commit 增长；可用的工作指标只能解释工程师间 38% 的消耗差异和周内 13% 的波动。
调查还发现，技术岗位员工在近九成的智能体使用场景中把 agent 用于编码之外的任务，如分析、研究和规划。研究结论是 token 消耗衡量的是 agent 使用的广度而非工作成果与价值，建议企业仅将其用于资源核算，改用任务和结果层面的证据来评估 agent 辅助的工作。
The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures
Organizations use engineers' token spend to monitor AI adoption, set budgets, and measure productivity, but the metric's relationship with engineering output is unclear. We link five weeks of token usage in coding agents to the commits of 404 engineers at JetBrains, and analyze a company survey of 412 employees. We find that token spend varied mostly between engineers, and which engineers spent the most stayed largely the same over the five weeks. Among weeks with similar commit counts, the 90th percentile of spend was 39 to 240 times the 10th. Within an engineer, doubling spend was associated with about 10% more commits. Available work metrics explained 38% of the variation in spend between engineers and 13% of the week-to-week variation within an engineer. In the survey, respondents in technical roles reported in nearly nine of ten agent evaluations using agents for tasks beyond coding, such as analysis, research, and planning. Token spend therefore measures how much engineers use agents across many kinds of work and not the outcome and value of that work. We recommend using token spend at most for resource accounting, and evaluating agent-assisted work with evidence about tasks and outcomes.
This report is categorized under AITOP's "Papers" desk, focusing on Europe's AI governance and technology ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include JetBrains, coding agent, token 消耗, 研发效能.
Source attribution: curated from arXiv cs.AI (@Shyam Agarwal, Olga Lvova, Agnia Sergeyuk, Anastasiia Serova, Timofey Bryksin, Thomas Zimmermann, Bogdan Vasilescu).
