预注册审计发现25个认知架构模块中18个计算结果未进入实际输出
Real Computation or Merely Naming? A Pre-Registered Mechanism Audit of 25 Cognitive-Architecture Modules in an LLM-Based Agent System
研究者真刀真枪拆了一个号称有25个认知机制的Agent系统，结果18个模块算完根本没人用，还给了可复用的审计方法。
一项预注册研究对某 LLM 智能体系统宣称的 25 个认知机制模块进行静态审计，采用带行号证据和调用链追踪的四问题协议，对 11 个模块预注册可证伪预测并执行干预。结果显示 25 个模块中有 18 个的计算值从未到达任何生产输出，并归纳出 connected-but-never-called、called-but-inert、called-but-content-blind 三种新的空闲接线形态。跨 12 次干预的 61 条冻结判据中，53 条确认、5 条证伪。两例修复实验将规划器任务成功率从 0.028 提升到 1.000，状态控制器错误率降低 52%。
Real Computation or Merely Naming? A Pre-Registered Mechanism Audit of 25 Cognitive-Architecture Modules in an LLM-Based Agent System
Agent systems built on local large language models are increasingly marketed through the vocabulary of cognitive architecture: modules named global workspace, theory of mind, causal reasoning, or active inference are listed as features, and the presence of the name is taken as the presence of the mechanism. We ask the prior question -- does the named computation actually happen -- and contribute a transferable verification method for the claim-implementation gap: a four-question static audit protocol with line-number evidence and call-chain tracing, demonstrated on two systems. We audit a system claiming 25 cognitive mechanisms, pre-register falsifiable predictions for 11 modules and execute the interventions, replicate the protocol on the IIT-phi core of a second, independent system with its own pre-committed falsification criteria, and run two repair experiments as positive controls. In 18 of 25 modules the computed value never reaches any production output; three idle-wiring shapes new to the dead-code and code-smell literatures emerge (connected-but-never-called, called-but-inert, called-but-content-blind). Of 61 frozen criteria across 12 interventions, 53 were confirmed, two partially adjudicated, five falsified, one adjudication-ambiguous. Repairs raised planner task success from 0.028 to 1.000 and cut state-controller error by 52%. Module names and module computations must be audited as independent claims; the protocol and the five-class failure taxonomy are reusable for the claim-implementation gap at two-system, case-level depth.
