论文提出 Effective Entropy：用有效性感知熵修正 RLVR 探索多样性
The Illusion of Diversity: Aligning LLM Exploration via Effective Entropy
做 RLVR 训练的人该看看这篇。它指出 token 熵高其实是模型在乱说话，不是在探索，然后给了 VEPO 这个修法，在 MATH500 等六个数学基准上比 GRPO 准确率更高，还用统计检验验证了策略确实更多样。
一篇新论文指出，当前 RLVR（可验证奖励强化学习）训练普遍依赖 token 级或全局序列级的“盲”熵指标，这类指标无法区分有效推理与幻觉，高熵往往意味着混乱退化而非有价值探索，作者称之为“多样性的幻觉”。论文提出 Effective Entropy（有效熵），只在有效子空间内量化多样性，过滤掉错误轨迹的噪声，并据此实现 VEPO（Validity-aware Entropy Policy Optimization）方法，将其直接引入 RLVR 训练。VEPO 会产生自适应吸引动态，提升欠探索有效轨迹的概率，匹配大规模自回归模型序列概率极小的实际场景。
实验在六个数学推理基准上使用 1.5B 和 7B 的 DeepSeek-R1-Distill-Qwen 模型进行，聚合准确率一致超过 GRPO，重复采样成功率也具竞争力。在 MATH500 上以匹配正确数的直接评估显示，模型能解出更多不同的解题策略，结果由配对统计检验和三次种子训练对比支持。
The Illusion of Diversity: Aligning LLM Exploration via Effective Entropy
Reinforcement learning with verifiable rewards has shifted the alignment paradigm of large language models from subjective preference to objective correctness, yet a fundamental disconnect persists between exploration diversity and reasoning validity. Existing strategies predominantly rely on "blind" entropy metrics, e.g., token-level or global sequence-level entropy, which structurally fail to distinguish between effective reasoning and potential hallucinations. This misalignment creates an illusion of diversity, where high entropy tends to signal chaotic degeneration rather than valuable exploration. To bridge this gap, we introduce Effective Entropy, a validity-aware entropy concept that quantifies diversity exclusively within the valid subspace and systematically filters out the noise of incorrect trajectories. Built on this concept, we instantiate VEPO, a simple Validity-aware Entropy Policy Optimization method that directly incorporates effective entropy into RLVR training. VEPO induces an adaptive attraction dynamic that drives up the probability of under-explored valid trajectories, a mechanism that matches the practical regime of large-scale autoregressive models where sequence probabilities remain extremely small. Experiments on six mathematical reasoning benchmarks with 1.5B and 7B DeepSeek-R1-Distill-Qwen models show consistent aggregate accuracy gains over GRPO and competitive repeated-sampling success. A direct evaluation with matched correct counts finds more distinct strategies on commonly solved MATH500 problems, supported by paired statistical tests and three-seed training comparisons.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include RLVR, Effective Entropy, VEPO, GRPO.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.
