Research map / Reinforcement learning / RL from feedback

Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ADRPO, which scales the divergence regularization coefficient per sample by its advantage (beta0 - A), applied to W2-regularized flow matching for SD3 text-to-image, KL-regularized GRPO for LLMs, and Qwen2.5-Omni audio reasoning.

Problem

Fixed divergence regularization coefficients in RL fine-tuning of generative models force a trade-off between preserving capabilities/diversity and reward optimization, causing either limited reward gains or instability and reward hacking.

Approach

Make the regularization coefficient sample-dependent via beta_tot = beta0 - A, so high-advantage samples get weaker divergence penalties and low-advantage samples get stronger ones; combined with advantage-weighted flow matching (signed advantages) and advantage clipping.

Key results

• On SD3 text-to-image, ADRPO reaches ClipScore 32.97 and ImageReward 1.61, exceeding FLUX.1-Dev (12B) and SANA-1.5 (4.8B) on all reported metrics.
• On MMAU, Qwen2.5-Omni-7B fine-tuned with ADRPO reaches 76.0% total accuracy versus 74.2% for GRPO, 71.6% for Gemini 2.5 Pro, and 62.5% for GPT-4o Audio.
• Clipping-range ablation on MMAU shows all ADRPO settings beat GRPO, with total accuracy varying under 0.4%.

Measured effects reported in the paper

Surprising findings

• A 2B SD3 model fine-tuned with ADRPO outperforms the 12B FLUX.1-Dev and 4.8B SANA-1.5 on all reported metrics.
• On Qwen3 the authors report ADRPO reaches a final checkpoint with 5x higher reward than GRPO, with a three-phase exploration pattern.

Limitations

• Experiments limited to moderate-scale models (SD3 2B, Qwen2-0.5B, Qwen3-0.6B) due to compute constraints.
• Advantage estimation uses batch statistics, which may be inadequate for high-variance reward landscapes.
• Advantage-based formulation adds slight computational overhead versus reward-weighted methods.

Experimental setup

Datasets: DrawBench RAFT style prompts RLHFlow/test_generation_2k AVQA MMAU

Benchmarks: DrawBench MMAU

Metrics: ClipScore ClipDiversity Aesthetic BLIPScore ImageReward PickScore accuracy (sound, music, speech, total) reward-entropy trade-off

Baselines: DPO ORW-CFM-W2 RAFT GRPO FLUX.1-Dev SANA-1.5 Gemini 2.5 Pro GPT-4o Audio

Builds on: GRPO ORW-CFM-W2 DPO RAFT conditional flow matching

Closest papers by meaning

• Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization (ICLR 2025 · similarity 0.89)
• MaskSense: Confronting the Visual Exploration Trap in Masked Image Generation (NeurIPS 2026 · similarity 0.87)
• Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · similarity 0.86)
• Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Image Generation (NeurIPS 2026 · similarity 0.86)
• WTF?! Simulation-Free Reinforcement Learning with Wasserstein-Tilted Flow Maps (NeurIPS 2026 · similarity 0.86)
• David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.86)