Research map / Large language models / Reasoning and chain of thought
Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic comparison of GRPO (on-policy) and DPO (off-policy) for autoregressive image generation with Janus-Pro, studying in-domain vs out-of-domain generalization, reward model choice, and three scaling strategies.
Problem
How GRPO and DPO compare for chain-of-thought-style RL in autoregressive image generation, given domain-specific issues like in-domain vs out-of-domain prompts, learned reward models, and scaling choices.
Approach
First comprehensive comparison of GRPO and DPO on an autoregressive image generator, using identical prompts and matched compute, measuring ID (T2I-CompBench) and OOD (GenEval) performance across reward models and scaling strategies, and linking RM intrinsic generalization (best-of-4) to RL generalization.
Key results
- DPO beats GRPO in-domain on T2I-CompBench across reward models, averaging 11.53% higher; peak 7.8% gain with metric reward.
- GRPO generalizes better OOD on GenEval, averaging 1.14% higher than DPO; with HPS reward the peak gain is 2.42%.
- DPO is more sensitive to reward model choice on GenEval (range 2.72 vs GRPO's 1.81), and reward-model best-of-4 ranking (Unified > ImageReward > HPS) matches RL OOD ranking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DPO rather than GRPO | Janus-Pro with the same reward model · T2I-CompBench | Average score: DPO's in-domain performance surpassing GRPO by an average of 11.53% | Helped |
| use GRPO rather than DPO | Janus-Pro with the same reward model · GenEval | Overall score: surpassing DPO by an average of 1.14% | Helped |
| use Unified Reward instead of HPS | GRPO · GenEval | Overall score: 79.18 -> 80.99 | Helped |
| use Unified Reward instead of HPS | DPO · GenEval | Overall score: 77.31 -> 80.03 | Helped |
| double in-domain training data (ablation) | GRPO · T2I-CompBench | Average score: 50.49 -> 50.67 | Helped |
| triple in-domain training data (ablation) | GRPO · GenEval | Overall score: 79.18 -> 78.21 | Hurt |
| increase sampled images per prompt from 4 to 8 (ablation) | GRPO · GenEval | Overall score: 79.18 -> 80.29 | Helped |
| increase sampled images per prompt from 4 to 12 (ablation) | GRPO · GenEval | Overall score: 79.18 -> 78.99 | Hurt |
| double in-domain training data (ablation) | DPO · T2I-CompBench | Average score: 53.90 -> 54.12 | Helped |
| double in-domain training data (ablation) | DPO · GenEval | Overall score: 77.31 -> 78.88 | Helped |
| increase sampled images per prompt from 4 to 8 (ablation) | DPO · T2I-CompBench | Average score: 53.90 -> 55.17 | Helped |
| add two iterative DPO training cycles (ablation) | DPO · T2I-CompBench | Average score: 53.90 -> 55.19 | Helped |
Surprising findings
- Reward models' intrinsic generalization (best-of-4 selection) predicts the OOD generalization of the RL-trained policy.
- Excessive sampling or triple data scaling for GRPO causes slight OOD decline, and DPO-Iter degrades OOD generalization after two iterations.
Limitations
- Single baseline model (Janus-Pro) and a single pair of benchmarks limit generality.
- Compute matching is approximate, measured in GPU hours on 8 A100s.
- Scaling-data prompts generated by GPT-4o, and the generated prompt set is not stated as released.
Experimental setup
Datasets: T2I-CompBench GenEval
Benchmarks: T2I-CompBench GenEval
Metrics: T2I-CompBench attribute/relationship/complex scores GenEval overall and sub-category accuracy best-of-4 reward model accuracy
Baselines: Janus-Pro GRPO DPO DPO-Iter GRPO-Iter
Builds on: DPO GRPO Image Generation with CoT Janus-Pro Iterative DPO
Closest papers by meaning
- T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT (NeurIPS 2025 · similarity 0.91)
- Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation (NeurIPS 2025 · similarity 0.91)
- Let's Verify and Reinforce Image Generation Step by Step (CVPR 2025 · similarity 0.91)
- Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models (NeurIPS 2026 · similarity 0.90)
- GoT-R1: Unleashing Reasoning Capability of Autoregressive Visual Generation with Reinforcement Learning (ICLR 2026 · similarity 0.90)
- NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.