Habituation at the Gate: 人类审查者对AI智能体代码的审查力度随时间下降

Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code

精选理由

这篇论文用真实数据告诉你,人类审查AI代码时会越来越松懈——批准率涨了,评论却少了。做AI代码审核的团队应该看看。

AI 摘要

一项基于AIDev数据集的长达七个月的纵向分析(400名重复审查者,共11,429条审查记录)发现,审查者对AI生成代码的批准率从30.1%上升至36.8%(Wilcoxon符号秩检验p<10^{-6})。随经验增加,批准率累计差距达14.5个百分点。与此同时,行内评论量下降22%(p=0.0014),但审查延迟增加3.5倍。这种模式提示审查者可能因工作负荷而产生习惯性麻木,而非理性信任调整。

原文 · arXiv: OpenAI

Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code

As AI coding agents (e.g., GitHub Copilot, Devin, OpenAI Codex, Cursor) submit pull requests to open-source repositories at scale, a key question arises: do human reviewers gradually lower their scrutiny for AI-generated code over time? We conduct a longitudinal within-reviewer analysis using the AIDev dataset, studying 400 repeat reviewers who collectively submitted 11,429 reviews over a seven-month observation period. Comparing each reviewer's early and late review episodes, we observe a population-level shift in approval rate from 30.1% to 36.8% (Wilcoxon signed-rank p < 10^{-6} on paired shifts). Pooled by within-reviewer experience decile, the cumulative gap reaches +14.5 pp from first to tenth decile. This shift is experience-driven (persists after controlling for calendar time), agent-specific (human PR approval rates decline over the same period), and not explained by PR difficulty (median PR size is flat). However, review latency increases rather than decreases (+3.5x), while inline comment volume decreases (-22%, p=0.0014), suggesting reviewers spend more time in queue but less time actively inspecting code. The combination of rising approval, declining comment effort, and increasing queue time is most consistent with reflexive habituation under growing workload rather than rational trust calibration alone.