AI 研究智能体写论文：八成错误数字来自张冠李戴而非造假
Beyond fabrication: Misattributed numbers in papers written by AI research agents
八成错误数字不是编的，是从自己别的实验里抄错的，论文还给了个 90% 精确率的自动核查工具，做 AI 科研的都该看看。
arXiv 论文重新分析了某 AI 研究智能体基准的全部 540 条判定，并人工标注了 18 份提交中的 256 个数值声明。结果显示约五分之四的无依据数字并非凭空捏造，而是取自智能体自己文件里错误实验、指标或运行的数据。作者提出确定性声明-证据校验器，在规则预先固定的条件下达到 90% 以上精确率（85 个中 78 个正确，Cohen's kappa = 0.89）。对 99 份提交的应用还显示不同智能体的可追溯性差异明显，LLM 校验器虽能找回更多可追溯声明，但误判错误归因的数量约是它的三倍。
Beyond fabrication: Misattributed numbers in papers written by AI research agents
Verifying numerical claims in research written by AI agents is an important but challenging problem. Recent benchmarks show that many agent-written papers fail integrity checks, but such verdicts, often produced by a single LLM judge, identify failure without explaining its nature, and existing checks either test a paper for internal consistency or test whether a reported value appears somewhere in an agent's output. We re-analyze all 540 released judgments from a recent public benchmark of AI research agents (nine agents, fifteen open research problems) and manually label 256 numerical claims from 18 submissions. Failures concentrate on alignment between the paper and the underlying work rather than overt cheating, and about four in five unsupported claims correspond to values in the agent's own files that come from the wrong experiment, metric, or run. We introduce a deterministic claim-to-evidence checker that scores whether the surrounding context identifies the claimed result and calibrates each claim against numerical decoys. With its rules fixed before labeling, it certifies claims with over 90% precision against blinded annotators (78 of 85; Cohen's kappa = 0.89), while verifying only a subset of claims; most claims it cannot verify are nevertheless traceable, so failure to verify is not evidence of falsehood. An LLM verifier given the same evidence recovers more traceable claims but accepts about three times as many misattributed ones, so the two are best used together. Applied to 99 submissions, the checker reveals large differences in traceability across agents, and a second judge from a different vendor preserves the official agent ranking while passing twice as many papers. These findings motivate provenance-first auditing of AI-produced research.
