Research map / Image generation / Text-to-image diffusion
Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QDHF, which learns diversity metrics online from human (or proxy) similarity judgments via a linear latent projection trained with triplet contrastive loss, and plugs them into MAP-Elites for diversity-driven optimization across robotics, maze RL, and text-to-image latent search.
Problem
Quality Diversity algorithms need manually crafted diversity metrics, which are hard to specify for open-ended tasks; unsupervised alternatives (AURORA) capture variance in current solutions rather than diversity useful for discovering novel solutions.
Approach
Learn diversity metrics as a linear projection of features, aligned to human 2AFC similarity judgments via triplet contrastive loss, and progressively re-fine-tune this projection on the evolving QD archive during search, rather than learning it offline once.
Key results
- On the robotic arm task, QDHF reaches all-solutions QD score 72.5 vs 45.9 for best AURORA variant and 74.8 for QD with ground-truth metrics.
- On LSI singular prompts, QDHF gets CLIP score 68.47 vs 67.64 for SD Best-of-n, with mean pairwise DreamSim distance 0.517 vs 0.405.
- In a blind MTurk study (n=50), QDHF was preferred 58.4% vs 31.9% for SD Best-of-n on singular prompts and judged correct 52.8% vs 40.6% on compositional prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use progressively fine-tuned feedback-aligned diversity metrics instead of learning diversity metrics offline | QDHF vs QDHF-Base · Robotic arm, all solutions | QD score: 54.5 \pm 4.3 -> 72.5 \pm 0.9 | Helped |
| Use progressively fine-tuned feedback-aligned diversity metrics instead of learning diversity metrics offline | QDHF vs QDHF-Base · Maze navigation, all solutions | QD score: 35.6 \pm 0.6 -> 42.0 \pm 1.7 | Helped |
| Use progressively fine-tuned feedback-aligned diversity metrics instead of learning diversity metrics offline | QDHF vs QDHF-Base · Maze navigation, archive solutions | QD score: 23.7 \pm 0.9 -> 22.5 \pm 1.3 | Hurt |
| Optimize image latents with QDHF rather than selecting top-CLIP images from random latents | Stable Diffusion · LSI, singular prompts | Mean PD: 0.405 -> 0.517 | Helped |
| Optimize image latents with QDHF rather than selecting top-CLIP images from random latents | Stable Diffusion · LSI, singular prompts | CLIP score: 67.64 -> 68.47 | Helped |
| Sample latents distant from previously sampled latents | SD (Best-of-n+) · LSI, singular prompts | Mean PD: 0.405 -> 0.412 | No clear effect |
| Optimize image latents with QDHF rather than selecting top-CLIP images from random latents | Stable Diffusion · LSI, singular prompts | User preference: 31.9\% -> 58.4\% | Helped |
| Optimize image latents with QDHF rather than selecting top-CLIP images from random latents | Stable Diffusion · LSI, compositional prompts | Judgment on correctness: 40.6\% -> 52.8\% | Helped |
| Flip 5% of feedback labels (ablation) | QDHF · Robotic arm, all solutions | QD score: 72.5 \pm 0.9 -> 72.4 \pm 0.8 | No clear effect |
| Flip 10% or 20% of feedback labels (ablation) | QDHF · Robotic arm, all solutions | QD score: 72.5 \pm 0.9 -> 68.3 \pm 4.7; 62.6 \pm 5.1 | Hurt |
| Flip 20% of feedback labels (ablation) | QDHF · Robotic arm, all solutions | QD score: 53.0 \pm 9.4 -> 62.6 \pm 5.1 | Helped |
Surprising findings
- Best-of-n+ diversity heuristic did not improve diversity over plain best-of-n, suggesting a strong inductive bias in the diffusion model.
- For some singular prompts, more than half of users preferred the less diverse baseline outputs, showing diversity and user preference can diverge.
Limitations
- Diversity metrics learned by DreamSim-based preference model may not generalize (e.g., day vs night cityscapes showed little apparent diversity).
- Human-judgment simulation for structured tasks uses ground-truth distances, not real human labels; LSI feedback is sourced from DreamSim rather than humans.
- Diversity does not always align with user preference, and the paper's user study covers a limited number of prompts.
Experimental setup
Datasets: NIGHTS MCC-250 COCO prompts ImageNet prompts
Benchmarks: Robotic arm (inverse kinematics) Kheperax maze navigation LSI text-to-image (singular and compositional prompts)
Metrics: QD score coverage CLIP score DreamSim mean pairwise distance user preference perceived diversity correctness judgment
Baselines: AURORA-Pre (AE) AURORA-Inc (AE) AURORA-Pre (PCA) AURORA-Inc (PCA) QDHF-Base QD-GT SD (Best-of-n) SD (Best-of-n+)
Builds on: MAP-Elites AURORA RUDA DreamSim RLHF
Closest papers by meaning
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.89)
- Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Image Generation (NeurIPS 2026 · similarity 0.89)
- Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
- Behavioral Foundation Models for Quality Diversity (NeurIPS 2026 · similarity 0.88)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.