精选理由
想高效训练智能体模型？SAO用单轨迹采样打补丁，稳定训练1000步，在编码和推理基准上全面超越GRPO，干货论文。
提出Single-rollout Asynchronous Optimization (SAO)方法，解决异步RL中GRPO框架不适用于智能体任务的稳定性和离策略问题。用单轨迹采样替代组采样，配合价值模型训练和双面token级裁剪，在SWE-Bench Verified、BeyondAIME、IMOAnswerBench上持续优于GRPO及其变体。SAO已成功部署于GLM-5.2模型(750B-A40B)的智能体RL训练管线。
AI 翻译 · 中文
提出Single-rollout Asynchronous Optimization (SAO)方法，解决异步RL中GRPO框架不适用于智能体任务的稳定性和离策略问题。用单轨迹采样替代组采样，配合价值模型训练和双面token级裁剪，在SWE-Bench Verified、BeyondAIME、IMOAnswerBench上持续优于GRPO及其变体。SAO已成功部署于GLM-5.2模型(750B-A40B)的智能体RL训练管线。
Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-h…