TCPO把验证器分数拆成每一步的信用，让模型知道哪个回合真正有用，在数学、代码和Agent任务上都有效果。
TCPO是一种面向验证器引导多轮强化学习的回合级信用分配方法，将验证器分数转换为回合级优势。它通过参考比较构建三种信用估计：回溯信用衡量相对最优状态的即时进展，事后延迟信用识别有后续回报的非改进回合，选择性反事实估计修正高意外回合。在数学推理、代码生成和AppWorld智能体任务上的实验显示，TCPO在Qwen3-4B和DeepSeek-R1-Distill-Llama-8B上取得最佳或并列最佳的最优回合Pass@8，并减少成功所需轮次。
TCPO: Turn-Level Credit Policy Optimization
Verifier-guided reinforcement learning has become a powerful paradigm for improving LLM reasoning. In multi-turn settings, models receive a verifier score after each turn and iteratively refine their outputs. Although such scores provide dense feedback, they do not directly provide dense credit: a score measures the quality of the current output, while credit should measure how the current turn changes the refinement trajectory. We propose TCPO, a turn-level credit assignment method for verifier-guided multi-turn RL. TCPO casts credit assignment as score-to-credit conversion and constructs turn-level advantages through reference-based comparisons: retrospective credit captures immediate progress and regression relative to the best prior state; hindsight delayed credit identifies non-improving turns with later payoff; and selective fixed-history counterfactual estimation refines high-surprisal turns under the same history. Experiments on math reasoning, code generation, and AppWorld agent tasks show that TCPO improves or matches the strongest baselines across model scales, task domains, and verifier types. TCPO achieves the best or tied-best best-turn Pass@8 on Qwen3-4B and DeepSeek-R1-Distill-Llama-8B, reduces turns to success, and improves multi-turn agent performance. These results highlight score-to-credit conversion as a central ingredient for verifier-guided multi-turn policy optimization.