Research map / Image generation / Text-to-image diffusion
Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FiFA, an automated filter that selects a small subset (about 0.5%) of human feedback pairs for DPO fine-tuning of SD1.5 and SDXL, scoring pairs by proxy-reward margin, LLM text quality, and kNN text diversity.
Problem
Fine-tuning text-to-image diffusion models with human feedback via DPO is slow and expensive because large, noisy preference datasets like Pick-a-Pic v2 contain many low-margin, low-quality, or harmful pairs.
Approach
Score each preference pair by a combination of proxy reward margin (from PickScore/HPSv2), GPT-3.5 text quality score, and a kNN entropy-based diversity term, then keep the top-K pairs as an approximate solution to a subset-selection objective motivated by G-optimal design.
Key results
- SDXL trained with FiFA (about 5k pairs, 17.2 GPU hours) outperforms the full-dataset SDXL-DPO on HPSv2 metrics, with human annotators preferring FiFA outputs 54.8% vs 37.0% on overall quality.
- SD1.5 with FiFA uses 13.6 GPU hours vs 56.2 for full data and scores higher on PickScore (21.64 vs 21.19) on the Pick-a-Pic test set.
- FiFA reduces harmfulness rate for neutral prompts versus full-dataset training, where full-data training increased harmfulness by at least 30%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Filter training pairs with FiFA instead of using the full Pick-a-Pic v2 dataset | SD1.5 DPO · Pick-a-Pic test | PickScore: 21.19 -> 21.64 | Helped |
| Filter training pairs with FiFA instead of using the full Pick-a-Pic v2 dataset | SDXL DPO · HPSv2 benchmark (Pick trainset) | PickScore: 23.10 -> 23.17 | Helped |
| Filter training pairs with FiFA instead of using the full HPSv2 dataset | SD1.5 DPO · HPSv2 benchmark | PickScore: 21.05 -> 20.98 | Hurt |
| Filter training pairs with FiFA instead of using the full HPSv2 dataset | SDXL DPO · Pick-a-Pic test | PickScore: 22.32 -> 22.24 | Hurt |
| Filter training pairs with FiFA instead of using the full Pick-a-Pic v2 dataset | SDXL DPO · HPSv2 human evaluation | overall quality preference: 37.0% -> 54.8% | Helped |
| Use FiFA-selected data rather than the full dataset | SD1.5 and SDXL DPO · Pick-a-Pic v2 and HPSv2 training | GPU hours: 56.2 -> 13.6; 1760.4 -> 18.3 | Helped |
| Add high preference-margin selection (ablation) | FiFA data selection · Pick-a-Pic test | PickScore | Helped |
| Select low-margin rather than high-margin pairs (ablation) | FiFA data selection · Pick-a-Pic test | PickScore | Hurt |
| Use text quality selection alone (ablation) | FiFA data selection · Pick-a-Pic test | PickScore | No clear effect |
| Use text diversity selection alone (ablation) | FiFA data selection · Pick-a-Pic test | PickScore | No clear effect |
| Add text quality and diversity to high-margin selection (ablation) | FiFA data selection · Pick-a-Pic test | PickScore | Helped |
| Increase selected data count beyond 5,000 pairs (ablation) | FiFA · Pick-a-Pic test | PickScore | Hurt |
Surprising findings
- Training on the full Pick-a-Pic v2 dataset increased the harmfulness of generated images by at least 30% relative to the pretrained model.
- Selecting only high-reward-margin pairs without diversity or quality terms underperforms, and random or high-reward-only selection also underperforms.
Limitations
- Validated mainly with the DPO objective, not policy-gradient RLHF or online DPO variants.
- Performance depends on the original dataset quality; larger K can hurt when data is noisy.
- Harmfulness is assessed on only three neutral prompts with manual labels by the authors.
Experimental setup
Datasets: Pick-a-Pic v2 HPSv2 training set PartiPrompts HPSv2 benchmark
Benchmarks: Pick-a-Pic test set PartiPrompts HPSv2 benchmark
Metrics: PickScore HPSv2 reward LAION Aesthetic Score human preference win rate harmfulness rate
Baselines: Diffusion-DPO full dataset training coreset selection loss-based selection random selection high reward selection SD1.5-DPO released checkpoint
Builds on: Diffusion-DPO Direct Preference Optimization ASK-LLM conservative DPO
Closest papers by meaning
- Diffusion Model Alignment Using Direct Preference Optimization (CVPR 2024 · similarity 0.91)
- Aligning Diffusion Models by Optimizing Human Utility (NeurIPS 2024 · similarity 0.91)
- Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation (NeurIPS 2024 · similarity 0.90)
- Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback (NeurIPS 2025 · similarity 0.90)
- InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment (CVPR 2025 · similarity 0.89)
- David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.