N-GRPO 解决了 GRPO 框架中探索与语义保持的冲突，做强化学习或数学推理优化的研究者可以直接参考其嵌入混合策略。
当前大语言模型在数学推理中面临 rollout 阶段的两难：token 级采样产生冗余轨迹，嵌入级随机噪声破坏语义一致性。N-GRPO 提出语义邻居混合机制，通过混合锚点 token 与其最近语义邻居的嵌入来注入多样性，同时保持局部语义流形。实验表明，该方法在 DeepSeek-R1-Distill-Qwen 系列模型上，数学推理基准一致优于强基线，并展现出良好的分布外泛化能力。
N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization
The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental trade-off: token-level sampling often yields redundant trajectories that differ only in rephrasing, while embedding-level methods utilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization (GRPO) framework. Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the local semantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines on math reasoning benchmarks but also exhibits robust generalization capabilities on out-of-distribution tasks.