PAC方法解决了LLM多任务RL中任务分配问题，结合优势信号和实际奖励增益，提升训练效率。
研究人员提出PAC方法，结合优势衍生的可学习性和近期奖励增益两个信号，用于大语言模型的多任务强化学习。在多级推理和多领域推理两种设置下，PAC仅需更少的采样步骤就能达到相当的验证分数，并最终超越随机采样和基于优势的课程基线。该方法通过贝叶斯汤普森采样控制器在GRPO训练过程中分配任务资源。
PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs
Reinforcement learning (RL) is used to improve the reasoning abilities of LLMs, while training data span heterogeneous tasks. However, most RL post-training pipelines rely on fixed or manually designed task mixtures, even though task usefulness changes as training progresses. Online curriculum methods often define learnability by update magnitude, ignoring whether the update translates into reward gains, which can misallocate rollout budget toward tasks with large but ineffective updates. We propose PAC, a Progress-Augmented Advantage Curriculum for multi-task RL of LLMs that combines two task-level signals: advantage-derived learnability, which measures the magnitude of the policy update a task can induce, and recent reward gains, which show whether those updates have improved task performance. A Bayesian Thompson Sampling controller uses these signals to allocate rollouts across tasks during GRPO training. We evaluate PAC under two settings: a multi-level reasoning setting and a multi-domain reasoning setting. PAC improves sample efficiency and final performance: it reaches comparable validation scores with fewer rollout steps and achieves higher final averages than random sampling and advantage-based curriculum baselines in both settings. These results show that jointly tracking advantage signals and actual reward gains yields an effective online curriculum for LLM post-training.