Theory Explains Why Unsupervised RLVR Collapses on Large Datasets
Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR
The paper turns a known failure mode of self-rewarding training into a predictive formula, giving practitioners a way to estimate safe dataset sizes before a run collapses. It also identifies two concrete mitigations, removing the shared coupling direction or adding a marginal-entropy bonus, that were validated on an actual Transformer. Researchers working on verifiable-reward training and anyone training reasoning models without labels will find the scaling laws directly applicable.
A new arXiv paper (2610.37553) explains why label-free RLVR, which rewards models for agreeing with their own majority vote, collapses when trained on many problems. The authors model the collapse as a collective instability: shared coupling across problems grows at a rate proportional to λ², independent of dataset size N, while per-problem sharpening slows by 1/N. This yields a critical problem count N_c ≈ ln(1/δ)/(λ²G), confirmed by simulations showing a sharp transition and N_c scaling as λ⁻². Experiments on a small Transformer show that removing the shared direction prevents collapse entirely, while a marginal-entropy bonus stabilizes training.
Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR
Label-free reinforcement learning with verifiable rewards (RLVR) rewards a language model for agreeing with its own majority vote. It improves reasoning at first but eventually collapses: the self-assessed reward goes to one while accuracy falls, often until the model gives the same answer to every problem. The collapse appears only when training uses many problems. We explain this dataset-size dependence as a collective instability. In a solvable many-problem model in which problems share parameters, a population whose problems prefer different answers amplifies any shared perturbation for as long as the population-averaged slope $\langle F'\rangle$ of the single-problem self-reinforcement drift $F$ is positive. This holds while problems are still uncertain about their answers. Per-problem sharpening, slowed by $1/N$ for $N$ training problems, races against the shared mode, which grows at a rate $\proptoλ^2$ independent of $N$, where $λ$ is the strength of the shared coupling. This yields a critical number of problems $N_c\simeq\ln(1/δ)/(λ^2G)$, where $δ$ seeds the shared mode and $G$ decreases with the initial confidence. Simulations of the model confirm the predictions: a sharp transition, $N_c\proptoλ^{-2}$ up to a logarithmic correction, a single critical amplification $λ^2NG$ across couplings, confidence profiles and training variants, a logarithmic dependence on batch size, and no collapse for confident initial models. A kernel version of the theory identifies a protective coupling (generalization of sharpening) and a marginal-entropy bonus that stabilizes the uniform collective mode. In label-free RL of a small Transformer, removing the shared direction prevents collapse entirely, restoring it causes collapse, and collapse grows with training-set size and base-model entropy.
