这论文提了两个新招PVF和TETHER，给LLM强化学习用的，比标准价值函数强，和GRPO差不多甚至更好，搞RL的可以看看。
论文提出两种互补策略改进价值函数强化学习：特权价值函数（PVF）在不偏置策略目标的前提下注入额外token级信号；TETHER则根据价值函数准确性自适应地在组相对基线和价值基线之间插值。在多个推理任务上，这两种策略均稳定优于标准价值函数基线，并与平均基线GRPO相比具有竞争力或更优。
Le Critique: Privileged Value Functions for LLM Reinforcement Learning
Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit. Training is also blocked by straggler rollouts, reducing throughput and increasing off-policyness. Learned value functions theoretically address both problems, providing token-level advantages without requiring large groups. However, additional infrastructure engineering challenges combined with the practical success of critic-free methods have made it difficult to justify their inclusion in RL pipelines. We propose two complementary strategies to improve the performance of value function RL: 1) Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; 2) TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy. Across several reasoning tasks, both strategies consistently improve over the standard value function baseline, and are competitive with or outperform mean-baseline GRPO.