研究:RL 后训练对扩散模型的增益可由固定激活向量解释
RL Post-Training in Diffusion Transformers Is Activation Steering
这篇论文很反直觉:用一组固定的激活向量就能复现 RL 微调九成左右的增益,参数量只有 LoRA 的六十五分之一。做扩散模型微调或评估的人应该看看,可能会改变你对 RL 后训练到底在学什么的判断。
一项 arXiv 研究提出,RL 后训练给扩散模型带来的提升,大部分可以用一组与输入无关的 steering 向量来复现。研究者在冻结的 SD3.5-M 基座模型的各个 block 输出上加固定向量,参数量仅 0.29M,比 RL 训练的 LoRA(18.8M)少 65 倍。
这些向量通过拟合公开 RL-tuned checkpoint 的速度预测得到,目标函数中没有奖励信号。虽然向量对所有 prompt、token 和采样 seed 都相同,但在六个 RL-tuned 图像模型上恢复了 90.0%–96.3% 的奖励增益,在两个视频模型上分别达到 89.8% 和 111.9%。将向量从零初始化并用在线 RL 训练后,仅凭向量就在 SD3.5-M 的文字渲染(OCR)和人类偏好(PickScore)上超过了基于权重的 RL。
作者据此推断,policy-gradient 后训练的大部分增益来自将预训练生成器导向其已有能力,而非学习新的输入相关变换。这一结论提示 RL 微调的效果空间可能远小于当前参数量所暗示的规模,对下游微调的成本估算有直接参考意义。
RL Post-Training in Diffusion Transformers Is Activation Steering
Reinforcement learning (RL) post-training substantially improves the visual quality, text rendering, and compositional accuracy of diffusion transformers. Because capabilities such as rendering requested text or composing multiple objects are inherently prompt-dependent, a natural hypothesis is that improving them requires learning complex, input-dependent transformations. To isolate how much of these gains an input-independent shift explains, we probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M). We fit them to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. Although the vectors are identical for every prompt, token, and sampling seed, they recover 90.0-96.3% of the reward gains of six RL-tuned image models, and 89.8% and 111.9% on two RL-tuned video models. Initialized at zero and trained with online RL, the vectors alone exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M. A constant shift thus suffices for most of the gain of policy-gradient RL post-training: without new input-dependent transformations, it steers the pretrained generator toward what it can already compute.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SD3.5-M, diffusion models, reinforcement learning, LoRA.
Source attribution: curated from arXiv cs.LG (@Tuna Han Salih Meral, Kaan Oktay, Alec Helbling, Atakan Kara, Duen Horng Chau, Pinar Yanardag).