arXiv 论文分析 21000 行全由 Claude 编写的代码库可靠性
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
有人让 Claude 独立写了两万多行 Python 代码，论文统计了错误率，还发现四五个回复里就有一个事实错误。
一篇 arXiv 论文发布了一个 21000 行 Python 工具的完整开发历史数据集，代码与测试全部由 Claude 编写，没有人工参与。
作者还开源了两个代码溯源追踪工具，以及针对指令意图、提交溯源、响应可靠性的三套分类体系。分析显示 14.
3% 的 AI 代码生成事件包含真实错误，后被 AI 编写的测试套件捕获；AI 的交互回复中约四分之一到五分之一包含至少一处事实性错误。
研究还发现编程智能体 CLI 指令与 IDE 聊天指令在类型上不同，前者更侧重理解、规划和咨询。
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Claude, 编程智能体, AI代码, 代码溯源.
Source attribution: curated from arXiv cs.AI (@Douglas Leith).
