Models9月5日 22:16

AI需CoT监控因不可靠

Gary Marcus@GaryMarcus

Post

View the post

Why we picked it

Gary Marcus谈AI可靠性问题，解释为什么需要CoT监控，以及真正的核心挑战是什么。

Gary Marcus指出，若AI足够可靠遵循指令，则无需CoT监控。当前AI系统存在可靠性问题，这是核心挑战。CoT监控成为必要手段，但并非根本解决方案。

Original · Gary Marcus

we wouldn’t need CoT monitoring if AI was trustworthy enough to follow instructions. but it isn’t. and that—not monitoring—is really the core problem.📊 2 ⚡

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include CoT, Gary Marcus.

Source attribution: curated from Gary Marcus (@GaryMarcus).

#CoT #Gary Marcus

Keep reading

Recent stories with the same tag or section

Papers BuildersSelected00:02

Sebastian Raschka on why optimizing chain-of-thought hurts monitorability

The finding that RL training alone, not process supervision, drives CoT controllability redirects where safety teams should focus their monitoring effort. Researchers building reasoning models and eval teams designing oversight need to track both no-CoT capability and controllability as training scales up. The pretraining-perplexity link gives labs a concrete metric to watch when weighing next pretraining runs against safety budgets.

#CoT #RL #reasoning models #AI safety

Official accountSebastian Raschka@rasbtSource

Papers Builders13:03

Chinese content首个思维链层级定理：更多 CoT 推理步数可证明带来更强计算能力

数学味很浓的一篇理论论文，首次证明了思维链步数是种基础计算资源，多用步数确实变强，不是玄学。

#CoT #reasoning models

arXiv cs.LG@Thanh Le et al. (3)Source

Papers Builders16:01

Chinese content神经符号结合方法让 LLM 精确求解 Zebra 逻辑谜题

把逻辑谜题交给 LLM 解析、再丢给精确覆盖求解器算答案，纯 CoT 解不好的题这条路能稳定做对，做求解器集成可以参考。

#neurosymbolic AI #LLM #CoT #logical reasoning

arXiv cs.AI@Paulius Skaisgiris et al. (6)Source

Papers PolicySelected14:40

Paper: First Tokens After Thinking Tag Limit CoT Controllability

The finding points to a cheap, practical lever: intervening at the very start of a reasoning trace can double a model's compliance with reasoning-style instructions, no retraining required. Practitioners building monitored or steerable CoT pipelines can try token prefilling or banned-opening lists right away. The split between add/restyle control and omission control also gives benchmark designers a concrete axis to test separately.

#Chain-of-Thought #CoT #reasoning models #AI safety

elvis@omarsar0Source

Papers Builders11:12

Chinese contentDA-CoT：训练免费难度感知思维链方法降低多模态模型过度思考

研究团队发现 CoT 用多了反而掉分，提出用注意力熵判断何时才需要思考，免训练就在推理分割基准上拿了 SOTA，还更省算力。

#CoT #MLLM #visual localization #attention

arXiv cs.AI@Chanseul Cho et al. (4)Source

Papers BuildersSelected01:30

Reasoning Models Control Chain-of-Thought via Self-Instructions

The placement of an instruction matters more than its wording: constraints injected into the reasoning channel as self-instructions succeed where prompt-level constraints fail. Developers building monitoring or steering pipelines for reasoning models can trigger these self-instructions directly from the user prompt, no reasoning-channel access required. The finding revises earlier controllability benchmarks that may have understated what current models can do.

#reasoning models #prompt engineering #CoT #papers

DAIR.AI@dair_aiSource