PapersSelected

Task Difficulty Shapes Whether Chain-of-Thought Causally Drives Answers

From Decorative to Load-Bearing: Task Difficulty Shapes the Causal Role of Chain-of-Thought

Why we picked it

The finding inverts an intuitive assumption about monitoring: on easy tasks models silently bypass their written reasoning, so a readable trace carries little signal exactly where oversight is easiest. Safety teams building CoT-based monitors should test load-bearingness on their hardest task distributions, since that is where errors propagate before intervention. Reasoning-specific RL is shown to suppress propagation, offering a practical lever.

Researchers introduce continuation-based causal testing, an ablation-patch intervention that corrupts one reasoning step and forces a model to continue from the broken chain.

Body

Testing Gemma-2-9B-IT, Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Qwen-7B on GSM8K, MMLU and BIG-Bench Hard shows CoT becomes load-bearing only on hard tasks.

Error propagation rises 16x from GSM8K to BBH multistep arithmetic, and a variance partition over 28,584 continuations attributes 98.

8% of explained deviance to task difficulty versus 0.8% to perturbation type.

A blind two-annotator study (n=500) reached perfect agreement (Cohen's kappa = 1.00) on the error-propagation label.

Linear probes can read these behavioral modes, but activation steering flips only about 25% of error-propagation cases at best.

Original · arXiv: DeepSeek

From Decorative to Load-Bearing: Task Difficulty Shapes the Causal Role of Chain-of-Thought

Chain-of-thought (CoT) monitoring is only meaningful if written reasoning causally constrains the answer. We introduce continuation-based causal testing, an ablation-patch intervention that perturbs one reasoning step, truncates the chain, and forces the model to continue from the corrupted prefix. It measures how load-bearing a CoT is for the final answer, a behavioral notion distinct from mechanistic faithfulness. Across Gemma-2-9B-IT, Llama-3.1-8B-Instruct, and DeepSeek-R1-Distill-Qwen-7B on GSM8K, MMLU, and BIG-Bench Hard, CoT load-bearingness tracks model-relative task difficulty: on easy tasks models silently bypass their own reasoning; on hard tasks they follow corrupted steps and propagate errors. A matched 2x2 analysis shows task difficulty dominates perturbation type: error propagation rises 16x from GSM8K to BBH multistep arithmetic, and a variance partition over 28,584 continuations attributes 98.8% of explained deviance to task difficulty versus 0.8% to perturbation type. Reasoning-specific RL suppresses error propagation and compresses the gradient. A four-variant judge-sensitivity analysis and blind two-annotator study (n=500) show the error-propagation vs. non-propagation label is invariant to judge prompt, with perfect inter-annotator agreement (Cohen's kappa = 1.00). This gradient creates a structural problem for CoT-based oversight and AI safety monitoring: where the trace is easy to read it carries little signal, and where it matters errors propagate before a monitor can intervene. Linear probes on hidden states separate silent bypass, self-correction, and error propagation, but additive activation steering provides limited causal control, flipping only about 25% of error-propagation cases at best. Behavioral mode is readable but not reliably controllable.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and AI regulation, compliance frameworks, and sovereign compute policy. Key entities and technical topics under continuous tracking include Chain-of-Thought, DeepSeek-R1, Gemma-2, GSM8K.

Primary source verification: published directly by arXiv: DeepSeek with first-party authority.