这篇论文解决了强化学习给LLM智能体分配奖励时的一个逻辑问题：相同意思的步骤因轨迹成败拿了相反信用。SCPO在ALFWorld和WebShop上跑分挺高，最难的步骤提升明显。
论文提出SCPO，一种价值无关的奖励塑造方法，通过对比同组内成功与失败轨迹的中间步骤，为失败步骤恢复正向信用。该方法解决了因轨迹最终结果不同导致语义相似的中间步骤获得相反信用的问题。在ALFWorld基准上，1.5B参数模型达到93.7%±4.1%成功率；在WebShop基准上达到74.8%±2.0%成功率，提升集中在最难的多步任务。
Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed. Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts. Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO), a value-free reward-shaping method that mitigates this inconsistency by recovering step-level credit from successful siblings in the same rollout group. Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling. On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines, reaching 93.7+/-4.1 percent success on ALFWorld and 74.8+/-2.0 percent on WebShop at 1.5B parameters, with gains concentrated on the hardest multi-step tasks.