论文精选75°

研究揭示思维链并非可靠监督通道:58% 步骤存在虚构延续

When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight Channel

精选理由

这项研究戳破了 CoT 推理过程忠实反映模型思考过程的假设,做 AI 安全、可解释性研究或依赖 CoT 审计的团队值得关注——它提醒我们,看起来合理的推理链条可能只是事后编造的故事。

AI 摘要

一项新研究系统性地检验了思维链(CoT)推理过程与最终答案形成时间之间的对齐程度。研究者提出了一个步骤级的检测-分类-比较框架,使用答案承诺代理、Patchscopes、调谐透镜探针和因果方向消融等方法,对九个模型和七个推理基准进行了分析。结果显示,潜在承诺与显式答案到达仅在平均 61.9% 的步骤上对齐,其中 58% 的不匹配事件表现为“虚构延续”——模型在答案已稳定后继续生成看似深思熟虑的文本。在架构匹配的 Qwen2.5 与 DeepSeek-R1-Distill 对比中,推理管线改变了失败组成而非整体对齐度。研究还发现,步骤级对齐度越低,CoT 的实用性反而越大,表明最受益于 CoT 的场景往往时间忠实性最差。截断实验和捐赠-破坏测试进一步表明,大量承诺后的文本对最终答案并非关键。

原文 · arXiv: DeepSeek

When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight Channel

Chain-of-thought (CoT) traces are increasingly used both to improve language model capability and to audit model behavior, implicitly assuming that the visible trace remains synchronized with the computation that determines the answer. We test this assumption with a step-level Detect-Classify-Compare framework built around an answer-commitment proxy that is cross-validated with Patchscopes, tuned-lens probes, and causal direction ablation. Across nine models and seven reasoning benchmarks, latent commitment and explicit answer arrival align on only 61.9% of steps on average. The dominant mismatch pattern is confabulated continuation: 58.0% of detected mismatch events occur after the answer-commitment proxy has already stabilized while the trace continues producing deliberative-looking text, and a vacuousness analysis shows that the committed answer does not change during these steps. In architecture-matched Qwen2.5/DeepSeek-R1-Distill comparisons, the reasoning pipeline changes failure composition more than aggregate alignment, most clearly at 32B where confabulated steps decrease as contradictory states increase. Lower step-level alignment is also associated with larger CoT utility, suggesting that the settings that benefit most from CoT are often the least temporally faithful. Paired truncation and a complementary donor-corruption test further indicate that much post-commitment text is not load-bearing for the final answer. These findings suggest that CoT can remain useful while still being an unreliable report of when the answer was formed.