Research map / Multi-agent systems and games / LLM agents

Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SummQ, an adversarial multi-agent framework where summary and quiz generators/reviewers iteratively refine long-document summaries, with an examinee agent answering quiz questions from the summary.

Problem

Long document summarization with LLMs suffers from information loss, factual inconsistency, and coherence problems.

Approach

Pairs summarization with a quizzing task: quiz generators and reviewers produce comprehension questions, and an examinee agent answers them from the summary, using the quiz failures as feedback to iteratively refine the summary within a multi-agent ensemble with drafting, aggregation, voting, and debate.

Key results

• SummQ (ensemble) reaches R-1 41.58 and BERTScore-F1 62.76 on MENSA, the best among reported methods, and R-1 44.62 on BookSum versus 42.80 for CachED.
• Human evaluation with five PhD judges on 20 NLP papers: SummQ ensemble preferred over GPT-4o and o3 at 88% and 82% winning rates (65% and 60% after length-matched rephrasing).
• Ablating the quizzing mechanism drops single-agent R-1 from 39.30 to 34.30 on MENSA.

Surprising findings

• On GovReport, SLED, CachED and U.former match or beat SummQ on some ROUGE metrics, suggesting task-specific fine-tuning remains competitive there.
• More iterations do not always help; performance peaks around 3 iterations and further refinement gives diminishing or negative returns.

Limitations

• Human-judged advantage partly reflects longer summaries (2084 vs 982-1450 words), a length bias only partly controlled by rephrasing.
• High cost: about $14.45 and 3.30M input tokens per instance for the ensemble variant.
• Results depend heavily on proprietary GPT-4o backbone and the LLM-as-judge is from the same model family.

Experimental setup

Datasets: MENSA BookSum GovReport

Benchmarks: MENSA BookSum GovReport

Metrics: ROUGE-1 ROUGE-2 ROUGE-L BERTScore-F1 LLM-as-a-Judge winning rate human winning rate

Baselines: TextRank LongT5 U.former SLED CachED GPT-4o GPT-4.1 GPT-5 o3 DeepSeek-R1 Qwen3-32B HM-SR

Builds on: multi-agent debate self-refinement

Closest papers by meaning

• Chain of Agents: Large Language Models Collaborating on Long-Context Tasks (NeurIPS 2024 · similarity 0.87)
• BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.84)
• SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.84)
• Learning from Failure: Improving Meeting Summarization without Good Samples (AAAI 2024 · similarity 0.84)
• Language Models of Code Are Few-Shot Planners and Reasoners for Multi-Document Summarization with Attribution (AAAI 2025 · similarity 0.84)
• NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens (ICLR 2025 · similarity 0.84)