做语言智能体强化学习的团队，可以用 PaW 在现有 RL 流程中零成本加入世界模型监督，提升智能体对环境的理解能力，值得在项目中尝试。
论文提出 PaW 框架，在强化学习训练语言智能体时，利用策略 rollout 中的动作-观测对作为世界模型监督信号，无需额外模拟器或推理计算。通过动作熵筛选数据、噪声容忍损失和自适应损失平衡三个组件，PaW 在多个智能体任务基准上显著优于纯 RL 基线。该方法解决了 RL 缺乏环境反馈监督的问题，让智能体不仅知道“做什么能得高分”，还理解“动作对环境的影响”。实验表明标准 RL rollout 即可提供有效的世界模型训练信号，降低了世界模型的应用门槛。
Policy and World Modeling Co-Training for Language Agents
Reinforcement learning (RL) improves large language model (LLM) agents by teaching them which actions lead to high rewards, but provides little supervision on what those actions do to the environment. World modeling (WM) can fill this gap, yet existing approaches often require separate simulators, extra training stages, or additional inference-time computation. We observe that on-policy RL rollouts already contain the needed signal: each transition pairs an action with its resulting next observation. Based on this observation, we propose PaW, a Policy and World modeling co-training framework that adds auxiliary WM supervision to the same policy during RL, without changing the inference paradigm. To make auxiliary WM supervision informative and stable, PaW introduces three components: action-entropy-based WM data selection, noise-tolerant WM loss, and reward-adaptive loss balancing. Experiments on three agentic task benchmarks show consistent improvements over strong RL baselines across models and RL algorithms. These results suggest that standard RL rollouts are a practical source of WM supervision for language-agent training.