SAFT用结构先验替代人工标注，给VLM奖励模型降噪，训练收敛更快，EPIC距离也更好了。
SAFT是一种自监督方法，通过LoRA适配器利用任务固有结构先验来正则化VLM的潜在空间，无需人工标注即可在线优化奖励信号。在多个基础模型能力范围内测试，SAFT持续降低奖励噪声，加速策略收敛，并使EPIC距离显著改善。研究表明VLM奖励模型的失败多源于结构脆弱性而非语义误解，SAFT为文本条件强化学习提供了可扩展的稳定路径。
Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning
Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL). Recent work uses large foundation Vision-Language Models (VLMs) as reward models, computing text-observation similarity to bypass manual reward engineering. Although promising, these rewards are often noisy and unreliable, limiting their direct utility during deployment. We present Structure-Aware Fine-Tuning (SAFT), a simple, self-supervised method that refines these imperfect reward signals online without access to ground-truth supervision. SAFT leverages intrinsic structural priors to regularize the VLM's latent space via LoRA adapters. We rigorously evaluate SAFT across a spectrum of base model capabilities to demonstrate its versatility. Our results show that SAFT consistently denoises the reward landscape, yielding faster policy convergence and substantially improved alignment (EPIC distance) relative to the underlying base model, suggesting that failures can often be attributed to structural brittleness rather than semantic misunderstanding. By replacing extensive human preference annotation with structural inductive biases inherent to the task, SAFT offers a scalable path for stabilizing text-conditioned RL and underscores the broader value of incorporating task structure as a general inductive bias.