论文揭示 LLM 智能体「价值-策略解耦」：内部算出最优动作却不采用
Represented but Not Used: Value-Policy Dissociation in LLM Agents
这篇论文做了一件很细的活：拿 Qwen2.5 和 GPT-OSS 在 Grid 和 WebShop 上证明模型内部其实能算出哪个动作更优，但策略层面偏偏不用。对做智能体可靠性或可解释性研究的人来说是很有用的诊断框架，NVT 这种干预思路也值得上手复现。
arXiv 论文《Represented but Not Used: Value-Policy Dissociation in LLM Agents》研究了 Qwen2.5-7B-Instruct、Qwen2.5-32B-Instruct 和 GPT-OSS-20B 三个模型在 Grid 部分可观测环境和 WebShop 模拟购物环境中的序贯决策机制。研究发现候选动作的相对价值可以从中间层隐藏状态中线性解码出来，甚至在策略选择了更差动作的状态上也是如此，作者将这种「内部表示了价值但策略不采用」的差距命名为 value-policy dissociation（价值-策略解耦）。
在干预实验部分，论文对比了沿价值读出方向的直接干预和 Natural Value Transport（NVT）——一种从隐藏状态随动作价值的自然变化中学到的变换。结果表明直接干预效果有限且不一致，而 NVT 能更稳定地影响动作偏好，尤其在较深网络层。论文还构建了一个闭环控制器，用价值读出作为传感器、NVT 作为执行器，观察到任务表现提升。
这一发现说明 LLM 智能体的失败未必源于「没算明白」，而可能是「算对了却不采纳」，为用干预手段直接改进智能体决策提供了机制层面的切入点。
Represented but Not Used: Value-Policy Dissociation in LLM Agents
Large language model (LLM) agents need to choose actions whose consequences unfold over multiple future steps, yet poor decisions do not reveal whether the model failed to represent those consequences or failed to use them. Mechanistic interpretability has studied internal representations of environment state and plans, but sequential decision making also requires evaluating how candidate actions change future outcomes. Reinforcement learning formalizes this as action value. We study this computation in Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and GPT-OSS-20B across a partially observed Grid environment and WebShop, a simulated web shopping environment. We find that the relative values of candidate actions are linearly decodable from intermediate hidden states, including at states where the policy prefers a worse action. We call this gap between value representation and policy choice value-policy dissociation. To investigate how represented value relates to action selection, we compare direct interventions along the value readout direction with Natural Value Transport (NVT), a transformation learned from hidden state changes that naturally result from differences in action value. While readout-direction interventions have limited and inconsistent effects, NVT more consistently influences action preferences, particularly at later layers. We also test a closed-loop controller that uses the value readout as a sensor and NVT as an actuator, observing improvements in task outcomes. These results suggest that LLM agents can represent useful prospective value without reliably translating it into policy choice.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Qwen2.5, GPT-OSS-20B, 智能体, 可解释性.
Source attribution: curated from arXiv cs.AI (@Yuzhang Luo, Mohamed Aghzal, Liangming Pan, Ziyu Yao). Cross-source corroboration: 2 independent sources have covered this development (including rohanpaul_ai), confirming industry-wide signal strength.
- rohanpaul_ai11-05 22:24原文
