研究提出延续式因果测试：CoT 在难题上才真正影响答案
From Decorative to Load-Bearing: Task Difficulty Shapes the Causal Role of Chain-of-Thought
拿 Gemma、Llama、DeepSeek 三个模型做了个挺狠的测试：简单题模型根本不看自己的推理，难题上错误会一路传到底，这对做 CoT 监控的人是个警告。
arXiv 论文提出 continuation-based causal testing，通过扰动一个推理步骤、截断链条并让模型续写，来衡量 CoT 对最终答案的因果作用。在 Gemma-2-9B-IT、Llama-3.1-8B-Instruct 和 DeepSeek-R1-Distill-Qwen-7B 上测试 GSM8K、MMLU 和 BIG-Bench Hard，结果显示 CoT 的承重程度与任务难度相关：简单任务上模型会绕过自己的推理，困难任务上则会跟随被污染的步骤传播错误。对 28,584 条续写的方差分解显示，98.8% 的可解释偏差来自任务难度，仅 0.8% 来自扰动类型，错误传播率从 GSM8K 到 BBH 多步算术上升 16 倍。盲评双人标注实验（n=500）的 Cohen's kappa 达 1.00，而激活转向最多只能纠正约 25% 的错误传播案例。
From Decorative to Load-Bearing: Task Difficulty Shapes the Causal Role of Chain-of-Thought
Chain-of-thought (CoT) monitoring is only meaningful if written reasoning causally constrains the answer. We introduce continuation-based causal testing, an ablation-patch intervention that perturbs one reasoning step, truncates the chain, and forces the model to continue from the corrupted prefix. It measures how load-bearing a CoT is for the final answer, a behavioral notion distinct from mechanistic faithfulness. Across Gemma-2-9B-IT, Llama-3.1-8B-Instruct, and DeepSeek-R1-Distill-Qwen-7B on GSM8K, MMLU, and BIG-Bench Hard, CoT load-bearingness tracks model-relative task difficulty: on easy tasks models silently bypass their own reasoning; on hard tasks they follow corrupted steps and propagate errors. A matched 2x2 analysis shows task difficulty dominates perturbation type: error propagation rises 16x from GSM8K to BBH multistep arithmetic, and a variance partition over 28,584 continuations attributes 98.8% of explained deviance to task difficulty versus 0.8% to perturbation type. Reasoning-specific RL suppresses error propagation and compresses the gradient. A four-variant judge-sensitivity analysis and blind two-annotator study (n=500) show the error-propagation vs. non-propagation label is invariant to judge prompt, with perfect inter-annotator agreement (Cohen's kappa = 1.00). This gradient creates a structural problem for CoT-based oversight and AI safety monitoring: where the trace is easy to read it carries little signal, and where it matters errors propagate before a monitor can intervene. Linear probes on hidden states separate silent bypass, self-correction, and error propagation, but additive activation steering provides limited causal control, flipping only about 25% of error-propagation cases at best. Behavioral mode is readable but not reliably controllable.
