Research map / Large language models / AI and society
Quality-Diversity through AI Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QDAIF, which combines MAP-Elites quality-diversity search with LLM-based mutation (LMX) and LLM-based AI feedback for both quality and diversity scoring, evaluated on creative writing (opinions, stories, poetry).
Problem
Quality-diversity search is hard to apply to qualitative text domains like creative writing because hand-designed quality and diversity measures are difficult to specify algorithmically.
Approach
Use language models both to generate variation (LMX few-shot crossover within MAP-Elites) and to score quality and diversity via natural-language yes/no or label log-probabilities, so the QD archive is filled using AI feedback instead of hand-coded measures.
Key results
- QDAIF achieves significantly higher QD score than all baselines in Opinions and Stories with mean bootstrapped 95% CIs across 5 seeds.
- On poetry with GPT-4, QDAIF reaches QD score 130 (CI 118-145) vs 76 for Random-Poems and 99 for Fixed Seed Rewrite.
- Human study: QDAIF human QD score 0.772 vs 0.767 for Fixed-Few-Shot; human-AI diversity label agreement 0.833.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use QDAIF rather than the text-generation baselines | creative writing generation · Opinions and Stories | QD score | Helped |
| use QDAIF rather than random poem generation | Poetry · Poetry | QD score: 76 (CI: 67 - 85) -> 130 (CI: 118 - 145) | Helped |
| use evolutionary search rather than rewriting only the seed poem (ablation) | QDAIF poetry generation · Poetry | QD score: 99 (CI: 72 - 117) -> 130 (CI: 118 - 145) | Helped |
| use QDAIF instead of Fixed-Few-Shot | creative writing generation · Opinions, Stories - Genre, Stories - Ending (averaged) | Human QD score: 0.767 -> 0.772 | No clear effect |
| use QDAIF instead of Fixed-Few-Shot | creative writing generation · Opinions, Stories - Genre, Stories - Ending (averaged) | Human quality rating: 4.133 -> 3.900 | Hurt |
| use zero-shot rather than seeded initialization (ablation) | QDAIF · Stories - Genre | Human QD score: 0.772 -> 0.383 | Hurt |
| replace LMX-Near with LMX-Replace mutation (ablation) | QDAIF with zero-shot initialization · Stories - Genre | Human QD score: 0.383 -> 0.540 | Helped |
| replace LMX-Near with LMX-Replace mutation (ablation) | QDAIF with seeded initialization · Stories - Genre | Human QD score: 0.772 -> 0.763 | No clear effect |
| increase LMX mutation model size from 13B to 70B (ablation) | QDAIF · Stories - Genre | Human quality rating: 3.43 (13B) -> 4.03 (70B) | Helped |
| increase LMX mutation model size from 13B to 70B (ablation) | QDAIF · Stories - Genre | QD score | No clear effect |
| increase AI feedback prompt examples from zero-shot to 4-shot (ablation) | QDAIF · Stories - Genre | Human quality rating: 3.43 -> 3.93 | Helped |
| increase AI feedback prompt examples from zero-shot to 8-shot (ablation) | QDAIF · Stories - Genre | Human quality rating: 3.43 -> 4.03 | Helped |
Surprising findings
- Pre-trained embedding-based feedback (QDEF) led to reward hacking, with human-rated quality lower than AI-feedback-based QDAIF.
- Quality-only LMX does not significantly outperform fixed few-shot prompting baselines in QD score.
- AI quality score correlation with human quality ratings drops at the highest fitness range (0.995 to 1).
Limitations
- AI feedback quality correlation with humans breaks down at highest fitness scores, suggesting reward hacking.
- Diversity axes must still be manually specified, so the method does not autonomously discover interesting diversity measures.
- Evaluated mainly on creative writing domains with a small human study (280 annotations).
Experimental setup
Datasets: hand-written few-shot seed examples
Benchmarks: Opinions Stories - Genre Stories - Ending Stories - Genre and Ending Poetry
Metrics: QD score coverage human QD score human quality rating (Likert) human-AI agreement human-human agreement
Baselines: Fixed-Few-Shot Shuffling-Few-Shot Random-Search LMX, Quality-Only LMX, ROUGE-L LMX, NSAIF Random-Poems Fixed Seed Rewrite Targeted-Poems QDEF (embedding feedback)
Builds on: MAP-Elites Evolution through Large Models (ELM) Language Model Crossover (LMX) OpenELM
Closest papers by meaning
- AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text (ICLR 2025 · similarity 0.87)
- Jointly Reinforcing Diversity and Quality in Language Model Generations (NeurIPS 2026 · similarity 0.87)
- Generative Monoculture in Large Language Models (ICLR 2025 · similarity 0.86)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.86)
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (NeurIPS 2025 · similarity 0.86)
- Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.