Papers

Shadow Posterior Matching:强化少步扩散模型的奖励对齐新方法

Reinforcing Few-Step Diffusion Models via Shadow Posterior Matching

Why we picked it

如果你在做少步扩散模型的 RL 调优,这篇值得看:SPM 用重加噪的 shadow posterior 拿奖励信号,比现有少步 RL 方法奖励更高、reward hacking 更少,还顺带解决对齐蒸馏 teacher 的老问题。做图像生成对齐的工程团队可以直接参考它的损失设计。

arXiv 论文 2611.25998 提出 Shadow Posterior Matching(SPM),针对 4-step 少步扩散模型的黑-box 奖励强化问题给出新方案。论文指出两个结构性难题:少步生成器是采样器而非 score 模型,其速度不等于生成分布的前向速度;同时它必须与不共享转移过程的未蒸馏 teacher 模型对齐。

Body

SPM 的做法是把生成样本重新加噪得到 shadow 输入,从 shadow posterior 中提取奖励 credit。作者从 KL-正则化奖励目标推导出一个回归损失,把奖励数值转化为奖励方向,同时把生成器输出锚定到 teacher,并让 shadow 输入与实际部署输入保持一致。

在 4-step 生成器实验中,SPM 相比已有的少步 RL 方法取得更高奖励,且 reward hacking 更少,绕开了传统方法把代价计入 reward credit 的缺陷。

Original · arXiv cs.LG

Reinforcing Few-Step Diffusion Models via Shadow Posterior Matching

Few-step diffusion models are widely deployed for their efficiency. However, reinforcing them with black-box rewards, as is routine for multi-step models, faces two structural challenges. First, a few-step generator is a sampler rather than a score model, so its velocity is not the forward velocity of the generated distribution. Second, it must be anchored to the undistilled teacher, which shares no transitions with it. Existing few-step RL methods handle the latter with a marginal KL to the teacher, but each pays for the former in its reward credit. We propose Shadow Posterior Matching (SPM), which renoises generated samples into shadow inputs and obtains the credit from their shadow posterior. From the KL-regularized reward objective, we derive a regression loss that turns reward values into a reward direction while anchoring the generator's outputs to the teacher and aligning the shadow inputs with the deployed inputs. Applied to 4-step generators, SPM achieves higher rewards with less reward hacking than prior few-step RL methods.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include diffusion models, reinforcement learning, SPM, reward hacking.

Source attribution: curated from arXiv cs.LG (@Jingqiao Xiu, Yicong Li).