Research map / Large language models / Retrieval and knowledge
Achieving Human Parity in Content-Grounded Datasets Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Genie, a three-stage (content preparation, few-shot LLM generation, reward-model and NLI filtering) pipeline that synthesizes large content-grounded datasets (wish-QA, wish-summarization, wish-IE) and shows models trained on them match or beat human-written data.
Problem
Content-grounded generation tasks (long-form QA, summarization, information extraction) lack high-quality training data, and human annotation is expensive and slow, while existing web-scraped datasets are noisy.
Approach
Generate task-specific examples from crawled or existing content passages via few-shot prompting of an LLM, then filter for format, faithfulness (T5-11B NLI model), and quality (DeBERTa-v3 reward model) to produce grounded synthetic data at scale.
Key results
- Flan-T5-XL trained on Wish-QA-NQ reaches ROUGE-L 28.2 on ASQA versus 10.5 for the base Flan-xl, and a reward of 30.3 on ELI5 matching gold.
- In a human Turing-style test, synthetic questions were judged human-written in 72% (ELI5), 63% (NQ), and 49% (ASQA) of cases.
- Filtering raised human-rated answer faithfulness from 53% to 76% and overall answer quality from 3.48 to 4.58 on a 5-point scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Flan-xl on WiFS synthetic QA instead of human ASQA data | Flan-xl · ASQA test | Reward: 68.6 -> 85.1 | Helped |
| Train Flan-xl on WiFS synthetic QA instead of human ELI5 data | Flan-xl · ELI5 test | ROUGE-L: 13.1 -> 13.8 | Helped |
| Train Flan-xl on Wish-Summarization instead of Flan-xl alone | Flan-xl · CNN-DailyMail test | Reward: 96.3 -> 97.5 | Helped |
| Train Flan-xl on Wish-Summarization instead of human CNN-DailyMail data | Flan-xl · CNN-DailyMail test | ROUGE-L: 33.3 -> 28.6 | Hurt |
| Train Flan-xl on synthetic medical QA rather than human ASQA data | Flan-xl · PubMed test | ANLI: 60.8 -> 81.8 | Helped |
| Train Flan-xl on Wish-IE rather than human Databricks data | Flan-xl · Information extraction test | Reward: 70.1 -> 87.4 | Helped |
| Add faithfulness and reward filtering to WiFS (ablation) | WiFS synthetic QA · WiFS human assessment | question relevance: 67% -> 92% | Helped |
| Add faithfulness and reward filtering to WiFS (ablation) | WiFS synthetic QA · WiFS human assessment | answer faithfulness: 53% -> 76% | Helped |
| Add faithfulness and reward filtering to WiFS (ablation) | WiFS synthetic QA · WiFS human assessment | overall answer quality: 3.48 -> 4.58 | Helped |
| Add faithfulness and reward filtering to WiFS (ablation) | WiFS synthetic QA · WiFS human assessment | question clarity: 63% -> 67% | Helped |
| Train on medical-domain synthetic QA rather than NQ synthetic QA | Flan-xl · PubMed test | ANLI: 73.1 -> 81.8 | Helped |
| Train on medical-domain synthetic QA rather than NQ synthetic QA | Flan-xl · PubMed test | Reward: 44.5 -> 39.4 | Hurt |
Surprising findings
- Out-of-domain synthetic data (Wish-QA-NQ) performed as well as or better than in-domain synthetic data, and Wish-QA-NQ and Wish-QA-MED achieved similar results on PubMed.
- Flan-xl achieved the highest faithfulness scores on ASQA/ELI5, attributed to its short, nearly extractive answers despite low performance.
Limitations
- Greedy decoding and a fixed reward threshold of 0.5 chosen by experimentation without systematic tuning.
- Faithfulness evaluation relies on automatic NLI and lexical metrics; human faithfulness judgments show residual unfaithful cases that are often subtle.
- Human evaluation uses a single expert annotator for the naturalness test on 100 question pairs.
Experimental setup
Datasets: ELI5 ASQA Natural Questions (NQ) CNN-DailyMail Wikipedia (crawled) PubMed Databricks dolly IE
Benchmarks: ASQA ELI5 CNN-DailyMail PubMed-QA Databricks IE
Metrics: ROUGE-L BERTScore F1 Reward model score K-Precision ANLI faithfulness vocd-D lexical diversity human Turing-style naturalness and quality assessment
Baselines: Flan-T5-XL ASQA-trained model ELI5-trained model CNN-DailyMail-trained model Databricks-dolly IE model
Builds on: Self-Instruct Unnatural Instructions q2d Open-Assistant reward model
Closest papers by meaning
- SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generation (ICLR 2025 · similarity 0.85)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.84)
- SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs (ICML 2025 · similarity 0.84)
- Putting People in LLMs’ Shoes: Generating Better Answers via Question Rewriter (AAAI 2025 · similarity 0.84)
- RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs (NeurIPS 2024 · similarity 0.83)
- MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning? (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.