做LLM推理强化学习的团队终于有了一个兼顾计算和样本效率的方案——BASIS用单次采样就达到多采样的效果，训练成本大幅降低，建议做RLHF或推理优化的开发者点开看看。
BASIS 是一种无需评论家的后训练算法，通过单次采样每个提示的轨迹，并利用整个批次中跨提示的信息共享来改进价值函数估计。实验表明，与单次采样的 REINFORCE++ 基线相比，BASIS 将价值函数估计的均方误差降低了 69%，且单次采样的 MSE 低于 8 次采样的组均值估计器。这种改进带来了更好的策略优化：BASIS 用更少的训练时间达到了接近多采样 GRPO 型基线的性能，并常优于单采样 REINFORCE 型基线。该工作解决了强化学习在计算效率与样本效率之间的权衡问题。
BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
Reinforcement learning with verifiable rewards has become a standard recipe for improving the reasoning abilities of large language models. Existing algorithms face a tradeoff between computational efficiency and sample efficiency in value estimation and policy learning. We introduce BASIS, a critic-free post-training algorithm designed to address this tradeoff. At each online training step, BASIS samples only one rollout per prompt, but leverages rich information across prompts in the entire batch to improve value function estimation. Our experiments demonstrate that BASIS reduces MSE in value function estimation by 69% compared to REINFORCE++, a representative single-rollout baseline, and achieves lower MSE with one rollout than group mean estimators with 8 rollouts. This improvement in value estimation translates to better policy optimization: using substantially less training time, BASIS achieves performance close to multi-rollout GRPO-type baselines and often outperforms single-rollout REINFORCE-type baselines.