Research map / Large language models / Retrieval and knowledge
QuerySum: A Multi-Document Query-Focused Summarization Dataset Augmented with Similar Query Clusters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QuerySum, a 27,041-sample multi-document query-focused summarization dataset with similar-query clusters, and a query-aware transformer model built on PEGASUS that sets a new benchmark.
Problem
Query-focused summarization research is limited by small or low-quality datasets that are either single-document, domain-specific, or dominated by factoid questions.
Approach
Builds a large multi-document QFS dataset from Answers.com queries, Google search answers, and Wikipedia source documents, with human-verified quality and clusters of similar queries annotated as synonymous, related, or unrelated; also proposes a query-aware encoder with semantic-based pooling and relevance-based inter-document attention.
Key results
- The proposed query-aware model reaches ROUGE-1 36.73, ROUGE-2 12.94, ROUGE-L 31.12 on QuerySum, versus 32.11 / 10.43 / 27.18 for QS-PEGASUS.
- Ablation: removing sentence-level attention drops ROUGE-1 from 36.73 to 35.44; removing word-level pooling drops it to 36.25.
- Adding the query improves BART ROUGE-1 by 1.61 on QuerySum, 1.04 on Debate standard, and 2.00 on AQUAMUSE, with near-zero gain on WikiHow (0.03).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed query-aware multi-document model instead of QS-PEGASUS | Query-focused summarization · QuerySum test | ROUGE-1: 32.11 -> 36.73 | Helped |
| Use the proposed query-aware multi-document model instead of QS-PEGASUS | Query-focused summarization · QuerySum test | ROUGE-2: 10.43 -> 12.94 | Helped |
| Use the proposed query-aware multi-document model instead of QS-PEGASUS | Query-focused summarization · QuerySum test | ROUGE-L: 27.18 -> 31.12 | Helped |
| Add the sentence-level component (ablation) | Our model · QuerySum test | ROUGE-1: 35.44 -> 36.73 | Helped |
| Add the sentence-level component (ablation) | Our model · QuerySum test | ROUGE-2: 12.46 -> 12.94 | Helped |
| Add the sentence-level component (ablation) | Our model · QuerySum test | ROUGE-L: 30.13 -> 31.12 | Helped |
| Add the word-level component (ablation) | Our model · QuerySum test | ROUGE-1: 36.25 -> 36.73 | Helped |
| Add the word-level component (ablation) | Our model · QuerySum test | ROUGE-2: 12.65 -> 12.94 | Helped |
| Add the word-level component (ablation) | Our model · QuerySum test | ROUGE-L: 30.74 -> 31.12 | Helped |
Surprising findings
- WikiHow shows almost no ROUGE gain from adding the query (0.03 ROUGE-1), suggesting it is poorly suited to query-aware summarization.
- The Debatepedia standard split has a data leakage problem, with over 70% of test summaries appearing or having near-duplicates in training.
Limitations
- Manual checking mostly removes bad samples rather than producing human-written summaries, so quality is below human-written datasets like CNN/DailyMail.
- Summaries are Google-extracted answer paragraphs and source documents are first paragraphs of Wikipedia pages, which limits realism.
- The way to use query clusters for QFS is left largely unexplored.
Experimental setup
Datasets: QuerySum AQUAMUSE DUC 2005-2007 Debatepedia WikiHow PubMedQA
Benchmarks: QuerySum Debatepedia WikiHow AQUAMUSE
Metrics: ROUGE-1 ROUGE-2 ROUGE-L
Baselines: Random Ext-oracle Transformer (CONCAT) Transformer (ADD) BERTSUM CTRLSum QS-BART QS-PEGASUS
Builds on: PEGASUS Transformer BERTSUM CTRLSum
Closest papers by meaning
- SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.85)
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis (NeurIPS 2024 · similarity 0.85)
- Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.84)
- RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (ICLR 2024 · similarity 0.84)
- PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models (NeurIPS 2024 · similarity 0.84)
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.