arXiv 论文:coding agent 做假设检验会让假阳性率飙到 98%
Inference after coding agents
这篇对做数据分析的人很关键:四个前沿 agent 在零效应数据上最高 98% 的概率给你“显著”结果。它还给了个可操作的重放纠偏方案,能把假阳性压回 5%。如果你让 agent 跑回归、做 A/B 分析,先看看这篇再信 p 值。
arXiv 新论文《Inference after coding agents》指出,用 AI coding agent 做实证研究中的假设检验,会带来一种新型选择性推断:agent 在隐式估计量空间里进行开放式搜索,且选择与估计过程可被程序化重放。实验显示,在真实效应为零的模拟数据上,四个前沿 agent 被要求“产出有说服力的发现”时,54% 到 98% 的情况会报告显著 p 值。
作者提出的纠偏办法是:在 agent 完成分析后,重置并在数据子样本上重放它,估计该 agent 在结果“看起来统计显著”时选中这一分析路径的相对概率。只要 agent 的选择行为稳定,修正后的检验在条件于选择的情形下渐近有效且一致。该修正从黑盒 agent 的实际选择中学习,不依赖对其搜索方式的建模,因此不受底层模型快速迭代影响。
模拟实验中,该方法把高假拒绝率压回接近 5% 的名义水平,作者还在两篇已发表论文的应用案例上做了演示。
Inference after coding agents
Empirical research is changing rapidly as AI coding agents are becoming increasingly proficient in every step of it, from writing code to manipulating data. We argue that while the potential of coding agents for science is vast, their use in hypothesis testing poses a substantial threat to classical inference, as these agents enable a supercharged form of selective inference. In particular, we argue that coding agents carrying out empirical research introduce a new type of post-selection inference that combines human-like, open-ended search over an implicit space of estimators with programmatic replayability of both the selection and the estimation process. Indeed, we find that on simulated data with zero effects, four frontier agents report significant p-values between 54% and 98% of the time when asked to produce a compelling finding. We propose a simple procedure for detecting and correcting these distortions. After an agent has completed its analysis, we reset and replay it on subsamples of the data to estimate the agent's relative probability of selecting that analysis when it looks statistically favorable. For agents with stable selection behavior, the resulting corrected test is asymptotically valid and consistent conditional on selection. Importantly, the correction is learned from the black-box agent's actual choices rather than a model of how it searches, so it remains valid even as AI technology continues to evolve rapidly. In simulations, the correction robustly lowers the high false rejection rates to near the nominal 5%. We illustrate the method on applications from two published papers.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include arXiv, coding agent, hypothesis testing.
Source attribution: curated from arXiv cs.AI (@Florian Gunsilius, David Van Dijcke, Kaspar Wüthrich).