PC-Dr.DPO：让多数投票伪标签的测试时强化学习更可靠
On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
用多数投票当伪标签做测试时强化学习容易翻车，这篇论文给出了误差分解和修复方法，做无标注数据适配的可以看看。
arXiv 论文分析了 TTRL 中多数投票伪标签的可靠性问题，将误差拆解为总体多数偏差和依赖提示的有限采样波动。作者提出 PC-Dr.DPO，根据观察到的投票可靠性对提示内稳健重加权进行条件化，使采样敏感提示的更新更保守，并用缓存机制缓解自监督遗忘。在 6 个数学与知识密集型推理基准、5 个模型骨干上，PC-Dr.DPO 在 30 个设置中的 24 个匹配或超过最强 TTRL 基线。
On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose \emph{Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO)}, which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.