Research map / Image generation / Text-to-image diffusion
Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds VisionPrefer, a 1.2M-preference, four-aspect dataset annotated by GPT-4V, trains the VP-Score reward model on it, and uses PPO (ReFL) and DPO (D3PO) to fine-tune Stable Diffusion v1.5 toward human preferences.
Problem
Human preference datasets for text-to-image alignment are expensive to build and cover few preference dimensions, limiting instruction tuning of open text-to-image models.
Approach
Use a multimodal LLM (GPT-4V) as a fine-grained preference annotator across four aspects (prompt-following, aesthetic, fidelity, harmlessness) with scores, rankings and textual rationales, then train a BLIP-based reward model on the averaged scores and use it for PPO and DPO fine-tuning.
Key results
- VP-Score achieves 70.46 average preference accuracy across ImageRewardDB, HPD v2 and Pick-a-Pic, second only to HPS v2 (71.32).
- VP-Score-tuned SD1.5 reaches Win+Tie above 50% against all compared reward models in PPO human study across three benchmarks.
- NSFW ratio on unsafe prompts drops to 4.4% for VP-Score fine-tuned models versus 21.1% for HPS v2 and 22.3% for PickScore.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the reward model on VisionPrefer | VP-Score · ImageRewardDB | preference prediction accuracy: 65.7 -> 66.3 | Helped |
| replace the BLIP backbone with CLIP (ablation) | VP-Score · ImageRewardDB, HPD v2, and Pick-a-Pic | preference prediction accuracy: 65.9 vs 66.3; 79.1 vs 79.4; 67.3 vs 67.1; Average 70.3 vs 70.5 | Mixed |
| remove Prompt-Following labels (ablation) | VisionPrefer-trained reward model for generation · DiffusionDB, Prompt-Following evaluation | WinRate: 58.07 -> 56.70 | Hurt |
| remove Aesthetic labels (ablation) | VisionPrefer-trained reward model for generation · DiffusionDB, Aesthetic evaluation | WinRate: 55.96 -> 53.82 | Hurt |
| remove Fidelity labels (ablation) | VisionPrefer-trained reward model for generation · Anything Prompts, Fidelity evaluation | WinRate: 52.20 -> 50.51 | Hurt |
| remove Harmlessness labels (ablation) | VisionPrefer-trained reward model for generation · Unsafe prompts | NSFW ratio: 4.4% -> 20.2% | Hurt |
| fine-tune with VP-Score rather than HPS v2 | Stable Diffusion v1.5 · Unsafe prompts | NSFW ratio: 21.1% -> 4.4% | Helped |
| use score feedback rather than rank feedback for GPT-4 V annotation (ablation) | VisionPrefer annotation · ImageRewardDB and HPD | preference prediction accuracy | Helped |
| increase GPT-4 V annotation temperature (ablation) | VisionPrefer annotation · ImageRewardDB and HPD | annotation accuracy | Hurt |
| increase the VisionPrefer training data size (ablation) | VP-Score · preference prediction | prediction accuracy | Helped |
Surprising findings
- Score-based feedback yields higher GPT-4V annotation accuracy than ranking-based feedback.
- Lower GPT-4V temperature improves annotation accuracy and consistency.
Limitations
- Hand and limb distortion persists; fidelity labels only mitigate it.
- Preference accuracy compared against human-annotated test sets with only about 70% human agreement as reference.
- Experiments use a single base model (Stable Diffusion v1.5) for fine-tuning.
Experimental setup
Datasets: VisionPrefer DiffusionDB ImageRewardDB HPD v2 Pick-a-Pic
Benchmarks: ImageRewardDB HPD v2 Pick-a-Pic DiffusionDB prompts Anything prompts
Metrics: preference prediction accuracy harmonic mean accuracy human win rate NSFW ratio
Baselines: CLIP ViT-H/14 Aesthetic predictor ImageReward HPS PickScore HPS v2 Pick-a-Pic HPD v2 ImageRewardDB
Builds on: ImageReward ReFL D3PO BLIP RLAIF
Closest papers by meaning
- Boosting Text-to-Video Generative Model with MLLMs Feedback (NeurIPS 2024 · similarity 0.90)
- Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Aligning Visual Contrastive learning models via Preference Optimization (ICLR 2025 · similarity 0.90)
- Preference Adaptive and Sequential Text-to-Image Generation (ICML 2025 · similarity 0.89)
- DreamAlign: Dynamic Text-to-3D Optimization with Human Preference Alignment (AAAI 2025 · similarity 0.89)
- Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.