λ-Controlled GRPO Stabilizes Flow-Matching RL for Image Generation
$λ$-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource
Compared with the strongest empirical stabilizer for Flow-GRPO, λ-Controlled GRPO raises OCR text accuracy and preference reward while keeping late-step path variance on budget, replacing per-step hand-tuned fixes with a single calibrated rule.
Flow-GRPO extends reinforcement learning to flow-matching image generators by treating the denoising sampler as a stochastic policy, but its importance ratios drift below one across denoising steps, clipping rates diverge, and fewer usable samples survive late in training. The paper traces these symptoms to a single per-step quantity, path variance, fixed exactly by the sampler's Gaussian transition kernel and cheap to estimate during training. Its λ-Controlled GRPO method calibrates importance-ratio behavior from this predicted law and allocates gradient effort across denoising steps by predicted cost, with both update scales set by standard policy choices rather than free tuning parameters. On a text-to-image model under two reward settings—difficult text rendering scored by optical character recognition and human preference matching scored by a preference model—λ-Controlled GRPO improves both metrics over the strongest empirical stabilizer and keeps late-step path variance within its intended budget.
$λ$-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource
Reinforcement learning is increasingly used to align image generators with reward signals, and Flow-GRPO recently extended this paradigm to flow-matching models by treating the denoising sampler as a stochastic policy that can be optimized from reward feedback. Training in this setting is unstable in a way specific to multi-step denoising: the policy update changes systematically across denoising steps, with importance ratios drifting below one, becoming increasingly dispersed, clipping at different rates, and leaving fewer usable samples late in training. Prior work treats these effects as separate failure modes and addresses each with a hand-tuned stabilizer. We show instead that they arise from a single per-step quantity, which we call path variance. This quantity is determined exactly by the sampler's Gaussian transition kernel and can be estimated cheaply during training. This reframes instability as a resource that can be measured and budgeted rather than a collection of symptoms to repair. Our method, $λ$-Controlled GRPO, calibrates importance-ratio behavior from this predicted law rather than from noisy empirical statistics, and allocates gradient effort across denoising steps according to their predicted cost. The two scales governing the update are fixed by standard policy choices rather than introduced as free tuning parameters. On a text-to-image model under two reward settings, rendering difficult target text scored by optical character recognition and matching human preferences scored by a preference model, $λ$-Controlled GRPO improves both text accuracy and preference reward over the strongest empirical stabilizer. It also keeps late-step path variance within its intended budget, precisely where the baseline systematically overshoots. The result is a Flow-GRPO update calibrated by its own transition law rather than stabilized after instability appears.