SpeCal 提出双层谱校准框架,改进无监督 RLVR 奖励可靠性
SpeCal: Spectral Calibration for Unsupervised RLVR
训练推理模型离不开带答案的数据,这篇提出只用模型自己的 rollout 就能校准奖励,靠隐藏状态的有效秩判断哪些样本值得学。做 RLVR 或无标签数据利用的研究者可以看下方法细节。
SpeCal 是一个针对无监督 RLVR 的双层谱校准框架,发表于 arXiv(编号 2611.20691)。其出发点是:RLVR 依赖标准答案或外部验证器,无法利用大量无标签推理数据,而现有无监督方法仅从解码结果和奖励统计构造监督信号,信息有限。
方法核心是用隐藏状态谱来评估监督可靠性。组(group)层面将谱质量与基于归一化熵有效秩的高斯带通门结合,筛选表示变化充分、有效维度处于合适区间的 rollout 组;rollout 层面用同一有效秩决定留一谱子空间中保留的主方向数量,并度量每个留出 rollout 的条件支持度。两个层面的信号对原有奖励产生的优势进行校准,而不重新定义无标签奖励本身。
对做推理模型训练的研究者来说,这条工作给出了一个不依赖标注答案的奖励校准思路,可作为无监督 RLVR 流水线中的即插即用信号模块。
SpeCal: Spectral Calibration for Unsupervised RLVR
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning abilities of large language models, but its dependence on ground-truth answers or external verifiers limits its ability to exploit abundant unlabeled reasoning data. Unsupervised RLVR alleviates this bottleneck by deriving scalar rewards from the model's own rollouts. However, the reliability of such self-generated supervision varies substantially across rollout groups and individual responses, while existing methods construct or refine their rewards mainly from decoded outcomes and reward statistics. These endpoint signals reveal what answers the model produces, but provide limited information about whether the underlying rollout representations form a reliable structure for learning. Our key observation is that hidden-state spectra offer a complementary view of supervision reliability by characterizing both the organization of a rollout group and the support received by each rollout within that group. Based on this observation, we propose SpeCal, a bi-level spectral calibration framework for unsupervised RLVR. At the group level, SpeCal combines spectral mass with a Gaussian band-pass gate over normalized entropy-based effective rank, favoring rollout groups with substantial representation variation whose effective dimensionality lies in a preferred regime. At the rollout level, the same group effective rank determines how many dominant directions are retained in a leave-one-out spectral subspace formed by the remaining rollouts, within which the conditional support of each held-out rollout is measured. The resulting signals calibrate the original reward-induced advantages at the group and rollout levels without redefining the underlying unlabeled reward.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include RLVR, SpeCal, reasoning models, unsupervised learning.
Source attribution: curated from arXiv cs.AI (@Kai Tang, Xing Zheng, Haobo Wang, Guangcheng Zhu, Shenzhi Yang, Zhengqing Zang, Yingfan Ma, Cheng Peng, Bowen Song, Weiqiang Wang).