GMTS方法让LLM推理训练更高效，前20%标记选择比传统熵方法表现更好。
研究人员提出梯度幅度标记选择(GMTS)方法，用于改进强化学习可验证奖励(RLVR)训练。实验显示，使用GMTS方法选择前20%的标记进行训练，在三个推理领域和多种模型尺寸下，均优于基于熵的标记选择方法。GMTS通过熵-梯度连接来近似梯度幅度排名，提供更精细的标记贡献估计。
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models' (LLMs) reasoning abilities, demonstrating remarkable effectiveness across reasoning tasks. Recent studies suggest that high-entropy tokens play an exceptionally important role in model training, since training with only the highest 20% entropy tokens yields significant performance gains. However, why such high-entropy tokens are beneficial remains insufficiently understood. In this work, we find that although high-entropy tokens within one answer tend to correlate with large gradient magnitude, entropy alone fails to consistently reflect token importance across different answers, considering the variations in the answer-level reward signals. Based on this observation, we introduce the Gradient Magnitude-based Token Selection (GMTS) method to quantify token importance, which leverages the entropy-gradient connection to approximate gradient-magnitude rankings for token selection. We find that training on the top 20% tokens ranked by GMTS consistently outperforms entropy-based token selection across three reasoning domains and various model sizes, suggesting that GMTS provides a more fine-grained estimate of token contribution for RLVR training.