Reward-guided Fine-Tuning via Wasserstein Gradient Flow

The method handles both non-differentiable and differentiable rewards without requiring reward gradients.Experiments on CIFAR-10 and ImageNet 256×256 datasets with diverse rewards show better reward alignment compared to baselines.

Reward-guided Fine-Tuning of One-Step Generative Models via Wasserstein Gradient Flow

To mitigate the time complexity of generative models, one-step generative models have recently emerged through direct mapping from noise to data in a single forward pass. However, the reward-guided fine-tuning method of one-step generative models remains largely unexplored. To address this, we consider one-step generators from an optimal transport view, investigating Wasserstein Gradient Flow (WGF) for modeling smooth and controlled distributional evolution in probability space. We then propose a novel reward-guided fine-tuning of a one-step generative model via WGF. We derive a practical training method that requires no reward gradients, thereby handling both non-differentiable and differentiable rewards. Moreover, our method provides smooth and stable reward-guided distributional updates while mitigating reward hacking and mode collapse. Experiments on 2D synthetic data, CIFAR-10, and ImageNet 256$\times$256 with diverse rewards, including JPEG (in)compressibility, class probability, Black-and-White and CLIP alignment, show that our method achieves better reward alignment compared to baselines.

This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include Wasserstein Gradient Flow, one-step generative models, reward-guided fine-tuning, ImageNet.

Source attribution: curated from arXiv cs.LG (@Hoseong Hwang, Woorim Han, Joungin Chun, Jinseong Park, Jaewoong Choi).