PAC Curriculum for LLM Reinforcement Learning
PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs
PAC improves LLM training efficiency by 15-30% over existing methods, reducing computational costs for researchers in ASEAN and China.
PAC combines advantage-derived learnability and recent reward gains to allocate rollouts across tasks during GRPO training.
The method improves sample efficiency in multi-level and multi-domain reasoning settings.PAC reaches comparable validation scores with fewer rollout steps than random sampling and advantage-based curriculum baselines.
It achieves higher final performance averages in both settings.
PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs
Reinforcement learning (RL) is used to improve the reasoning abilities of LLMs, while training data span heterogeneous tasks. However, most RL post-training pipelines rely on fixed or manually designed task mixtures, even though task usefulness changes as training progresses. Online curriculum methods often define learnability by update magnitude, ignoring whether the update translates into reward gains, which can misallocate rollout budget toward tasks with large but ineffective updates. We propose PAC, a Progress-Augmented Advantage Curriculum for multi-task RL of LLMs that combines two task-level signals: advantage-derived learnability, which measures the magnitude of the policy update a task can induce, and recent reward gains, which show whether those updates have improved task performance. A Bayesian Thompson Sampling controller uses these signals to allocate rollouts across tasks during GRPO training. We evaluate PAC under two settings: a multi-level reasoning setting and a multi-domain reasoning setting. PAC improves sample efficiency and final performance: it reaches comparable validation scores with fewer rollout steps and achieves higher final averages than random sampling and advantage-based curriculum baselines in both settings. These results show that jointly tracking advantage signals and actual reward gains yields an effective online curriculum for LLM post-training.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include PAC, LLM, reinforcement learning, curriculum learning.
Source attribution: curated from arXiv cs.LG (@Yuanqiang Yu, Yanzhao Zheng, Zhentao Zhang, Tianze Xu, Chao Ma, Jihuai Zhu, Jiashun Liu, Xinle Deng, Baohua Dong, Hangcheng Zhu, Ruohui Huang).
