RoCo-PRM：用 MoE 专家路由信号做过程监督，无需人工标注
A Strong MoE Is All You Need for Process Supervision
这篇论文挺有意思：不用人工标注，直接从 MoE 的专家路由统计里挖出每步推理的置信度信号，做成过程奖励模型，能自动标注还能跨模型迁移。
arXiv 论文提出从 MoE 模型的专家路由中提取过程监督信号。作者结合整体路由质量、头部路由质量和步内路由波动构建路由置信度，推出 RoCo-Rethink，定位置信度最低的推理步并选择性重新生成后续轨迹。RoCo-PRM 则用强 MoE 模型依据路由分数自动标注推理轨迹，再蒸馏成独立的 PRM。在多种 MoE 架构和多个推理任务上，两个方法均优于基线，RoCo-PRM 还能跨架构、跨任务迁移。
A Strong MoE Is All You Need for Process Supervision
Process reward models (PRMs) provide fine-grained supervision for multi-step reasoning, yet obtaining reliable process labels remains a central bottleneck. Existing approaches typically rely on human annotation, rollout-based estimation, or auxiliary verification. In this work, we explore a different source of process supervision natively exposed by Mixture-of-Experts (MoE) models: expert routing. We show that simple routing statistics provide an intrinsic process-confidence signal that is competitive with conventional output-logit confidence. Building on this observation, we construct routing confidence by combining overall routing mass, head routing mass, and within-step routing variation, and introduce RoCo-Rethink, which identifies the least confident reasoning step and selectively regenerates the subsequent trajectory. We further introduce RoCo-PRM, which uses a strong MoE to automatically annotate reasoning trajectories with routing-derived step scores and distills them into a standalone PRM. Across multiple MoE architectures and diverse reasoning tasks, both RoCo-Rethink and RoCo-PRM consistently outperform strong baselines, while RoCo-PRM transfers effectively across both model architectures and tasks. Our findings suggest that expert routing is more than a computational dispatch mechanism: it provides an architecture-native source of process supervision for reasoning.
