Sebastian Raschka on why optimizing chain-of-thought hurts monitorability
The finding that RL training alone, not process supervision, drives CoT controllability redirects where safety teams should focus their monitoring effort. Researchers building reasoning models and eval teams designing oversight need to track both no-CoT capability and controllability as training scales up. The pretraining-perplexity link gives labs a concrete metric to watch when weighing next pretraining runs against safety budgets.
A thread by Sebastian Raschka examines the tension in reasoning model development between training or optimizing chain-of-thought (CoT) and leaving CoTs untouched to preserve monitorability. Written CoTs are valuable because practitioners can inspect them to detect undesirable behaviour such as cheating, but explicitly optimizing CoTs risks eroding that visibility. Raschka frames two caveats: monitorability only works if the written reasoning reflects what the model actually does, and more capable models naturally write less in their CoTs, much like an advanced math student needs fewer notes during an exam.
The thread summarises empirical findings on measuring "harder to monitor" along two dimensions. No-CoT capability — how much reasoning a model can perform in a single forward pass without writing anything down — rises with better pretraining (lower perplexity), while architecture mostly does not matter. CoT controllability — how well a model can add, remove or change parts of its written reasoning when instructed — is raised substantially by reinforcement learning alone, and controllability is passed on through distillation.
Higher no-CoT capability and higher controllability both make models harder to monitor, confirming existing intuitions in the AI safety community. Raschka notes one apparent piece of good news: process supervision does not appear to be the main culprit behind reduced monitorability.
In reasoning model development, it's always been this interesting tension between training/optimizing chain-of-thoughts (CoTs) vs not touching them to preserve monitorability.
CoTs are cool because we can check what the model is up to (e.g., whether it cheats or not) as long as it spells out its thought process. And we risk losing that if we explicitly optimize CoTs.
Two caveats:
1) I.e., the catch of CoT and monitorability is that this only works if the written thinking reflects what the model is actually doing.
2) Also, when models become more capable, they spell out less in the CoT (like a cleverer math student needs less note-taking when solving problems during an exam).
Nothing new on that front, but an interesting thread below on how to measure "harder to monitor", with some interesting results that confirm the existing intuitions:
1) no-CoT capability (meaning how much reasoning the model can do in a single forward pass without writing anything down)
- Better pretraining (lower perplexity) raises this no-CoT capability - Architecture mostly doesn't matter 2) CoT controllability (meaning how well the model can change its written reasoning when asked to add, remove, or change parts of it)
- RL alone raises controllability a lot - Controllability is passed on through distillation
* Higher no-CoT capability and higher controllability both make monitoring harder.
PS: Perhaps the good news is that it seems that process supervision wasn't the main culprit.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include reasoning models, chain-of-thought, AI safety, Sebastian Raschka.
Source attribution: curated from Sebastian Raschka (@rasbt).
