做LLM推理强化学习的团队终于有了更精细的熵控制方案——PAEC在数学推理任务上直接提升多数投票性能，做RLVR的开发者值得关注这个位置感知的新思路。
论文提出PAEC（位置感知熵校准）方法，解决强化学习（RLVR）中策略熵过早崩溃的问题。传统全局熵正则化对所有位置均匀增加熵，在长推理轨迹中效率低下。PAEC通过局部top-p熵和top-two候选竞争构建软掩码，对决策敏感位置施加基于锚点的下界惩罚，防止这些位置的熵崩溃。在五个数学推理基准测试中，PAEC相比强RLVR基线提升了多数投票的宏平均性能，尤其在AIME类任务上增益明显。结果表明，推理RL中的熵管理应聚焦于决策敏感位置的选择性探索，而非均匀随机注入。
PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR
Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning but often suffers from rapid policy-entropy collapse, where the policy prematurely concentrates on narrow high-probability reasoning paths. While global entropy regularization can encourage exploration, uniformly increasing entropy across all token positions is inefficient for long reasoning trajectories, where many tokens are not decision-relevant. We propose Position-Aware Entropy Calibration (PAEC), a token-level entropy-management framework that constructs a soft mask from local top-p entropy and top-two candidate competition, and applies an anchor-based lower-bound penalty to prevent selected-position entropy collapse. Experiments on five mathematical reasoning benchmarks show that PAEC improves macro-average majority-vote performance over strong RLVR baselines, with clear gains on AIME-style tasks. Our results suggest that entropy management in reasoning RL should be formulated as selective exploration allocation over decision-sensitive positions rather than uniform randomness injection.