Research map / Reinforcement learning / RL from feedback

GARDO: Reinforcing Diffusion Models without Reward Hacking

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GARDO, a regularization framework for online RL fine-tuning of diffusion/flow models that gates KL penalties to high-uncertainty samples, adaptively resets the reference model, and reshapes positive advantages by DINOv3 diversity to mitigate reward hacking while keeping sample efficiency.

Problem

Online RL fine-tuning of text-to-image diffusion models with imperfect proxy rewards leads to reward hacking (proxy score rises while image quality and diversity fall), and standard KL regularization slows sample efficiency and restricts exploration.

Approach

Apply KL penalty only to the top ~10% of samples by reward uncertainty (estimated via win-rate disagreement with auxiliary reward models), periodically reset the KL reference model to the current policy, and multiplicatively scale positive advantages by a DINOv3 nearest-neighbor diversity score.

Key results

• On GenEval-trained SD3.5-M, GARDO raises diversity from 19.98 (Flow-GRPO beta=0.04 w/o div) to 24.95 and GenEval accuracy to 0.95 at 2000 steps.
• On OCR-trained SD3.5-M, GARDO reaches 0.91-0.96 OCR at 600-1400 steps while keeping Aesthetic (5.07) and ImageReward (0.92) at or above the reference model's levels.
• With DiffusionNFT, GARDO reaches 0.95 GenEval at 400 steps versus 0.72 for KL-regularized DiffusionNFT, while keeping Aesthetic at 5.23 vs 4.87.

Surprising findings

• Removing the standard deviation from GRPO advantage normalization alleviates reward hacking, since reward models give near-identical scores within groups and std normalization amplifies noise.
• Penalizing only about 10% of highest-uncertainty samples is sufficient to prevent reward hacking.

Limitations

• Depends on auxiliary reward models (Aesthetic and ImageReward) for uncertainty estimation.
• Scalability to resource-intensive video generation models is untested.
• Evaluation is limited to a few proxy rewards (GenEval, OCR, HPSv2 for Flux) and fixed prompt sets.

Experimental setup

Datasets: Flow-GRPO GenEval training set Flow-GRPO OCR training and test sets HPDv3 DrawBench

Benchmarks: GenEval OCR DrawBench Counting

Metrics: GenEval accuracy OCR accuracy Aesthetic score PickScore ImageReward ClipScore HPSv3 DINOv3 pairwise cosine diversity counting accuracy

Baselines: Flow-GRPO (beta=0) Flow-GRPO (beta=0.01) Flow-GRPO (beta=0.04) DiffusionNFT Pref-GRPO DDPO

Builds on: Flow-GRPO DiffusionNFT DDPO GRPO DPOK

Closest papers by meaning

• Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.92)
• David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.90)
• DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
• Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · similarity 0.89)
• Stitched Value Model for Diffusion Alignment (NeurIPS 2026 · similarity 0.88)
• Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL (NeurIPS 2026 · similarity 0.88)