ReMaRL:在冻结扩散语言模型中学习解掩码与重掩码策略
ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models
掩码扩散模型的解码顺序其实很影响准确率,这篇论文用 0.3M 参数的小策略在冻结的 LLaDA-8B-Instruct 上学怎么解掩码和重掩码,数学和代码都跑赢了启发式方法。做扩散语言模型推理优化的人值得看。
论文提出 ReMaRL(Remasking with Reinforcement Learning),一个约 0.3M 参数的轻量策略,用于决定掩码扩散语言模型中哪些 token 应解掩码、保留或重新掩码。该策略使用 GRPO(Group Relative Policy Optimization)从最终答案奖励直接训练,训练期间扩散模型保持冻结。
现有方法要么用启发式规则控制解掩码和重掩码,要么训练监督式重掩码器检测 token 错误,都无法直接以最终答案正确性为目标优化决策。ReMaRL 的关键点是允许重掩码:模型可以基于新生成的上下文修正早期预测,而现有的任务奖励解掩码策略只能单向解掩码。
在 LLaDA-8B-Instruct 上的实验显示,ReMaRL 在数学任务(4-32 次模型调用)和代码任务(8-16 次调用)的推理预算内取得了对比方法中的最高分数。
ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models
Masked diffusion language models generate tokens in parallel, and the order in which tokens are unmasked affects final-answer accuracy. Several methods learn unmasking policies from task rewards, but these policies cannot remask earlier predictions. Remasking lets the model correct those predictions using newly generated context. Heuristic rules guide both unmasking and remasking, while supervised remaskers learn to detect token errors. Neither approach directly optimizes these decisions for final-answer correctness. To address this gap, we introduce ReMaRL (Remasking with Reinforcement Learning), a lightweight policy that learns which tokens to unmask, retain, or remask. We train this roughly 0.3M-parameter policy with Group Relative Policy Optimization (GRPO) from final-answer rewards while keeping the diffusion model frozen. With LLaDA-8B-Instruct, ReMaRL achieves the highest scores among the compared unmasking and remasking policies within the tested inference budgets of 4-32 model calls on math and 8-16 calls on code.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include ReMaRL, LLaDA, GRPO, diffusion language models.
Source attribution: curated from arXiv cs.LG (@Egor Shibaev, Danila Biktimirov, Ivan Klimov, Artem Makoian, Viacheslav Meshchaninov, Dmitry Vetrov).