这篇论文提出RAIL，把干预选择当成在线学习问题，让模型在有限预算下学到更有效的rollout，适合做RL后训练的人看。
RAIL是一种训练时框架，通过在线上下文bandit建模干预选择，学习如何为策略生成rollout。它利用影子到实时程序收集干预轨迹，训练恢复控制器，使控制器随策略演化持续学习。在多个设置下，RAIL在有限rollout预算下持续提升性能。结果表明恢复感知干预能生成更有信息量且更少冗余的rollout，增强后训练的学习信号。
Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning
Critic-free group-based reinforcement learning has become a scalable approach for post-training large language models. However, most existing methods allocate the same number of rollouts to every task and trajectory state, even though some rollouts provide much more useful learning signals than others. Recent work has started to treat rollout generation as an adaptive decision, but two important limitations remain. First, intervention strategies are often based on fixed heuristics and therefore cannot adjust as the policy changes during training. Second, these methods usually decide only how many rollouts to generate, without explicitly controlling where and how to intervene. To address these limitations, we propose Recoverability-Aware Intervention Learning (RAIL), a training-time framework that learns how to generate rollouts based on the improvement produced by each intervention. RAIL models intervention selection as an online contextual-bandit problem and trains a recoverability controller using intervention traces collected through a shadow-to-live procedure. This allows the controller to keep learning while the underlying policy evolves. We evaluate RAIL in terms of effectiveness, adaptivity, expressiveness, and efficiency. Across multiple settings, RAIL consistently improves performance under limited rollout budgets. These results show that recoverability-aware intervention provides a principled way to generate more informative and less redundant rollouts, leading to stronger learning signals during post-training.