RLVR训练成本高昂，RELEX用极低成本实现同等推理提升，做LLM推理优化的团队可以直接用代码复现，值得一试。
研究发现强化学习（RLVR）训练中模型权重的变化轨迹是低秩且高度可预测的，大部分性能提升来自秩-1近似。基于此，研究者提出RELEX方法，仅需观察少量训练步数（如50步），通过线性回归外推未来检查点（如1000步），即可匹配甚至超越完整RLVR训练的性能。在Qwen2.5-Math-1.5B、Qwen3-4B-Base和Qwen3-8B-Base三个模型上验证，RELEX仅需15%的训练步数即可达到同等效果，且能外推至10-20倍于观察窗口。该方法无需额外学习模型，通过丢弃随机优化噪声实现去噪效果，从而提升外推性能。
You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories
Reinforcement learning with verifiable rewards (RLVR) has become a dominant paradigm for improving reasoning in large language models (LLMs), yet the underlying geometry of the resulting parameter trajectories remains underexplored. In this work, we demonstrate that RLVR weight trajectories are extremely low-rank and highly predictable. Specifically, we find that the majority of downstream performance gains are captured by a rank-1 approximation of the parameter deltas, where the magnitude of this projection evolves near-linearly with training steps. Motivated by this, we propose a simple and compute-efficient method RELEX (REinforcement Learning EXtrapolation), which estimates the rank-1 subspace from a short observation window and extrapolates future checkpoints via linear regression, with no learned model required. Across three models (i.e., Qwen2.5-Math-1.5B, Qwen3-4B-Base, and Qwen3-8B-Base), RELEX produces checkpoints that match or exceed RLVR performance on both in-domain and out-of-domain benchmarks, requiring as few as 15% steps of full RLVR training. Remarkably, RELEX is able to extrapolate far beyond the observation window at no training cost, predicting checkpoints up to 10-20$\times$ beyond the observed prefix with continued improvement (e.g., observe only the first 50 steps and extrapolate to 1000 steps). Our ablation analysis confirms the minimalist sufficiency of RELEX: neither increasing the subspace rank nor employing non-linear modeling yields further gains in extrapolation. Finally, we show that RELEX's success stems from a "denoising" effect: by projecting updates onto the rank-1 subspace, the model discards stochastic optimization noise that would otherwise degrade performance during extrapolation. Our code is available at https://github.com/weizhepei/RELEX.