Research map / Reinforcement learning / Policy gradient and actor-critic

Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Flow-DPPO, replacing PPO ratio clipping in online RL for flow matching models with an asymmetric mask based on the exact closed-form Gaussian KL divergence between old and new per-step policies.

Problem

Ratio clipping in Flow-GRPO-style RL for flow matching models is a noisy single-sample estimate of policy divergence, causing spurious clipping, bias in the ratio distribution, over-constraint and under-constraint, and instability under multi-epoch reuse.

Approach

Exploit the Gaussian per-step policy of flow samplers to compute the exact KL between old and new policies as a squared mean difference, and mask gradients only when the update moves further from the old policy and the divergence exceeds a threshold.

Key results

• On SD3.5-medium multi-reward, Flow-DPPO+CPS reaches 51.6 GenEval2 versus 47.8 for GRPO-Guard and 39.9 for Flow-GRPO.
• On FLUX2-9B multi-reward, Flow-DPPO reaches 57.7 GenEval2 versus 49.0 for GRPO-Guard and 46.8 for Flow-GRPO.
• Flow-DPPO variants show less out-of-domain degradation (PickScore, CLIP, HPSv2) during training and sustain gains under multi-epoch training (G64-I2, G32-I2) where baselines plateau or degrade.

Surprising findings

• Ratio clipping's log-ratio is dominated by noise: its standard deviation equals sqrt(2 KL), so even moderate KL causes spurious clipping.
• Retaining the full 2*sigma^2 normalization in the CPS KL (rather than the dropped-normalization convention of prior work) gives a tighter constraint that prevents distribution collapse.

Limitations

• Gains on out-of-domain and in-domain metrics are sometimes mixed; e.g., Flow-DPPO's CLIP score on FLUX2-9B single-table is lower than some baselines.
• Theoretical bound relies on maximum per-step TV divergence and reward magnitude, which are loose in practice; the predictive mask is only sketched as a heuristic proxy.
• Evaluation focused on GenEval2 and PickScore-related metrics with image models; video generation is only suggested as future application.

Experimental setup

Datasets: GenEval2 (20k synthetic training prompts) PickScore prompts

Benchmarks: GenEval2 PickScore CLIP score HPSv2

Metrics: GenEval2 score PickScore CLIP score HPSv2 KL divergence from pretrained model

Baselines: Flow-GRPO Flow-CPS GRPO-Guard Diffusion-NFT

Builds on: Flow-GRPO CPS DPPO TRPO PPO GRPO-Guard

Closest papers by meaning

• Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · similarity 0.91)
• Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.89)
• PRECISE: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models (NeurIPS 2026 · similarity 0.89)
• ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• On the Guidance of Flow Matching (ICML 2025 · similarity 0.88)
• Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization (ICLR 2025 · similarity 0.88)