这篇论文揭露了AI解释性评估的漏洞，并给出了RECAP方法，让你知道如何让模型内部信息真正可验证，而非被花言巧语欺骗。
论文指出自然语言自编码器通过重构分数评估解释的忠实性存在缺陷：在Qwen-2.5-7B的口语化器上，约2%的特定主张与重构相关，分数主要反映大意而非具体事实。在合成真值下，标准方法在5/5次运行中产生了共适应私有代码（重构依赖的虚假措辞）。提出两个审计协议（grounded-vs-true交叉和评估器交换）以及RECAP（通过协训练辅助预测器使指定内容可解码）。在RECAP训练的沙盒模型上，新口语化器真实陈述指定内容，代码消失，代价仅+0.001 nat。在预训练的Pythia-160M上，指定内容变得可探针解码，虽新口语化器仅部分传达（真值0.44-0.46 vs 近零对照）。独立探针对口语化器的真实主张评分高于虚假（AUC 0.96 vs 无RECAP 0.82），对抗攻击下RECAP探针仍能识别谎言（AUC 0.95），对照探针降至0.51。
Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations
Natural-language autoencoders score explanations of hidden activations by reconstruction: an explanation is deemed faithful if the activation can be regenerated from it. The test is structurally insensitive to individual false claims: if flipping a claim does not change the reconstruction, the claim is never penalized. We show the test is passed in two ways, neither faithful. On a released Qwen-2.5-7B verbalizer, explanations reconstruct well above chance while ~2% of specific claims are reconstruction-dependent, so the score tracks gist, not specific facts. Under exact synthetic ground truth, the standard recipe develops co-adapted private codes (false wording the reconstruction depends on) in 5/5 runs, and fixes that leave the target model unchanged do not help. We contribute two audit protocols, the grounded-vs-true cross and the evaluator swap, and RECAP (Readable Encodings via Co-trained Auxiliary Predictors): linear heads trained alongside the target model to keep designated content decodable. On RECAP-trained sandbox models, fresh verbalizers state the designated content truly and the codes vanish, at a +0.001-nat cost. This replicates on a pretrained Pythia-160M: the content becomes reliably probe-decodable, though a fresh verbalizer conveys it only in part (truth 0.44-0.46 vs a near-zero control). For interpretability, high reconstruction does not certify individual claims. For AI safety, RECAP makes designated internal content independently checkable against probes rather than asserted by prose a model can game: an independent probe scores the verbalizer's true claims above its false ones (AUC 0.96, vs 0.82 without RECAP). Against an adversary that edits an explanation to maximize the reconstruction score while lying (suppressing ~87% of its lie penalty), the RECAP probe still flags the lies (AUC 0.95) while the control probe collapses to chance (0.51).