AITOPAI热报
PapersSelected

Harvard and MIT paper traces reasoning control failures to "Okay" openers

rohanpaul_ai@rohanpaul_ai
Post
View the post
Why we picked it

The finding decouples chain-of-thought monitoring reliability from controllability benchmarks: a trivial token-level intervention accounts for much of the apparent control failure. Safety teams evaluating CoT faithfulness should report opening-word effects separately from information-hiding tasks. The paper spans 12 open models including Qwen3 and OLMo, so labs fine-tuning open-weight reasoners can reproduce the fix directly.

A new paper from Harvard and MIT researchers examines why reasoning models fail to control their chain-of-thought when instructed. Models like Qwen3-32B and OLMo-3.1-32B place about 99% of first-token probability on "Okay", anchoring their reasoning in a habitual opening. The authors found that banning "Okay" and "Alright" plus adding one worked example raised average compliance by 34 to 41 percentage points over instructions alone. Across 12 models, tasks that require hiding information responded less consistently, suggesting they represent a separate skill.

Original · rohanpaul_ai

New Harvard and MIT paper finds that reasoning models' poor control over their chain-of-thought partly comes from a habit of opening it with "Okay".

Safety teams read a model's reasoning to catch misbehavior, and benchmarks showing models can't steer that reasoning have been reassuring. Why models fail at this was unclear.

The authors told models like Qwen3-32B to reason in uppercase or number each sentence. Qwen3-32B and OLMo-3.1-32B put about 99% of first-token probability on "Okay", so the authors tried banning it or prefilling 1 compliant token.

Banning "Okay" and "Alright" and adding 1 worked example raised average compliance by 34 to 41 percentage points over instructions alone. Tasks that hide information, like avoiding a word, responded less consistently and look like a separate skill across 12 models.

If you trust chain-of-thought monitoring because of controllability scores, separate the opening-word problem from general control and report hiding tasks separately.

#Qwen3#OLMo#reasoning models#AI safety#chain-of-thought

Keep reading

Recent stories with the same tag or section
PapersBuildersMulti-sourceSelected14:19
DecisionRAG: Cornell paper replaces retrievers with a decision model over document trees

The interesting shift is that retrieval becomes a tree-traversal decision problem instead of an embedding-similarity problem, so latency scales with tree depth rather than corpus size, with billion-paragraph corpora theoretically needing around five decisions. Teams paying for a reranker stage, or struggling with weak first-stage recall on help centres and document corpora, can test the tree-plus-decision-model setup; on NQ a dense retrieval stage can optionally be mixed in to recover over-pruned evidence. It competes with Qwen3-Reranker-8B pipelines at lower cost.

#DecisionRAG#Jev#RAG#Qwen3
Event
shao__meng@shao__meng3 sourcesSource
PapersBuildersMulti-sourceSelected05:25
Cohere paper finds speculative decoding works well on MoE models

Practitioners running low-batch MoE inference, especially agent workloads with single-request traffic, can now justify benchmarking speculative decoding on their own hardware instead of ruling it out based on older cost models. The expert-reuse finding changes how inference engineers should estimate verification costs for open-weight models like Qwen3-235B on H100-class GPUs. Teams serving MoE models at low batch sizes should rerun their own latency tests before committing to a serving strategy.

#Cohere#Qwen3-235B#MoE#speculative decoding
Event
rohanpaul_ai@rohanpaul_ai4 sourcesSource