Papers

ROAM-GRPO:通过离模式策略挖掘提升 GRPO 推理多样性

ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

Why we picked it

做推理模型 RL 后训练的人可以看看这篇。它给 GRPO 加了个离模式挖掘机制,专治 SFT+RLVR 后输出千篇一律、pass@k 不涨的问题。在 MATH lvl5 上用 Qwen3.5-2b 验证,pass@16 提了 4.7%,多样性指标翻倍,方法可直接复现。

一篇新论文提出 ROAM-GRPO(Routed Off-mode Approach Mining for GRPO),针对 SFT 加 RLVR 后训练流程导致策略模式坍缩、输出同质化的问题,对 GRPO 算法进行增强,鼓励模型探索不同的推理策略。

Body

机制上,ROAM-GRPO 在训练中检测已被很好解决的问题,将其转入“离模式”通道,提示模型生成不同的解题策略,由 LLM judge 验证逻辑有效性与差异性,再自蒸馏回策略网络。实验在 MATH lvl5 数据集的 1000 题子集上训练 Qwen3.5-2b,峰值 pass@16 提升 4.7%,正确输出的嵌入余弦距离提升约 1.5-2 倍。

对做推理模型后训练的团队来说,这项工作给出了缓解 pass@k 精度收敛或退化、提升输出多样性的具体可复现方案。

Original · arXiv cs.AI

ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include GRPO, Qwen3.5-2b, RLVR, reasoning models.

Source attribution: curated from arXiv cs.AI (@Jeff Duan, Matt Thomson).