Research map / Image generation / Text-to-image diffusion
Rich Human Feedback for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Collects RichHF-18K, a dataset of rich human feedback (point-marked artifact and misalignment regions, misaligned prompt keywords, and four fine-grained scores) on 18K Pick-a-Pic images, and trains a multimodal transformer (RAHF) to predict it and use it to improve Muse and Latent Diffusion outputs.
Problem
Existing T2I evaluation and reward models reduce image quality to a single scalar or binary preference, and do not localize artifacts, misaligned regions, or misaligned words.
Approach
Enrich human feedback with region heatmaps for implausibility and misalignment, word-level misalignment labels, and fine-grained scores, then train one ViT+T5X multimodal transformer to predict all of these and apply the predictions to filter finetuning data, guide sampling, and inpaint flawed regions.
Key results
- Augmented-prompt model reaches PLCC 0.693 on plausibility and 0.580 on overall score, versus 0.495 and 0.337 for ResNet-50.
- Implausibility heatmap MSE on empty-GT samples drops to 0.00095 for augmented prompt, compared with 0.00093 for ResNet-50 (multi-head is worse at 0.00141).
- In human side-by-side evaluation, finetuned Muse was rated significantly or slightly better than original Muse in 51.83% of examples (21.5% significantly better plus 30.33% slightly better).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Muse using examples selected by predicted RAHF plausibility scores | Muse · 100 new prompts, human side-by-side plausibility evaluation | human preference: 21.5% significantly better; 30.33% slightly better; 31.33% about the same; 12.67% slightly worse; 4.17% significantly wo | Helped |
| Use task-augmented prompts instead of separate heads for plausibility prediction (ablation) | Our Model · RichHF-18K test set: plausibility | PLCC: 0.666 -> 0.693 | Helped |
| Use task-augmented prompts instead of separate heads for aesthetics prediction (ablation) | Our Model · RichHF-18K test set: aesthetics | PLCC: 0.605 -> 0.600 | Hurt |
| Use task-augmented prompts instead of separate heads for text-image alignment prediction (ablation) | Our Model · RichHF-18K test set: text-image alignment | PLCC: 0.487 -> 0.474 | Hurt |
| Use task-augmented prompts instead of separate heads for overall-score prediction (ablation) | Our Model · RichHF-18K test set: overall score | SRCC: 0.561 -> 0.562 | Helped |
| Use task-augmented prompts instead of separate heads for implausibility heatmap prediction (ablation) | Our Model · RichHF-18K test set: implausibility heatmap, all data | MSE: 0.01216 -> 0.00920 | Helped |
| Use task-augmented prompts instead of separate heads for text misalignment heatmap prediction (ablation) | Our Model · RichHF-18K test set: text misalignment heatmap, all data | MSE: 0.00303 -> 0.00304 | No clear effect |
| Use predicted implausibility heatmaps to mask and inpaint problematic regions, then select by predicted plausibility | Muse · Image region inpainting | image plausibility | Helped |
| Use the RAHF aesthetic score as classifier guidance | Latent Diffusion · Text-to-image generation | aesthetic quality | Helped |
Surprising findings
- The multi-head variant performs worse than ResNet-50 on implausibility heatmap prediction, while the augmented-prompt variant beats it, showing that task-specific prompting matters for multi-task heads.
- Misalignment heatmap prediction is consistently weaker than implausibility heatmap prediction, which the authors attribute to noisier, less well-defined misalignment labels.
Limitations
- Misalignment heatmap prediction is notably weaker than implausibility prediction, likely due to label noise and ambiguity for absent objects.
- Training data comes mainly from Pick-a-Pic, which is dominated by Stable Diffusion variants, so generalization to other generators is only shown for Muse.
- The authors note that using predicted heatmaps or misalignment sequences as RL reward signals was not explored.
Experimental setup
Datasets: Pick-a-Pic WebLI RichHF-18K
Benchmarks: RichHF-18K test set
Metrics: PLCC SRCC MSE CC KLD SIM NSS AUC-Judd token-level precision recall
Baselines: ResNet-50 PickScore CLIP CLIP gradient
Builds on: RLHF ImageReward PickScore Spotlight PaLI
Closest papers by meaning
- Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation (CVPR 2025 · similarity 0.91)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.91)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.91)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.90)
- Preference Adaptive and Sequential Text-to-Image Generation (ICML 2025 · similarity 0.89)
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.