Research map / Large language models / Language, safety and interpretability
Automatic Summarization of Long Documents (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes three training-free input-reduction pipelines (central truncation, document skimming with redundancy removal, and keyword-based segment selection) that let fixed-context LLMs summarize documents of over 70k words.
Problem
Transformer summarizers are limited by context size, so they cannot process long documents such as GovReport and BigPatent texts exceeding 70,000 words.
Approach
Reduce long documents to fit the context window without architectural changes, using middle truncation, uniform segment sampling with redundancy removal, and LDA-keyword-guided segment selection via sentence-transformer similarity, then summarize with an off-the-shelf LLM.
Key results
- On GovReport, summarization with keyword extraction using GPT-3.5 Turbo reaches R-1 61.99 and BERTScore 86.20, versus 66.0 BERTScore for BART with Unlimiformer.
- Central truncation with LongT5 achieves R-L 38.27 and BERTScore 82.19 on GovReport, and skimming with post-sampling removal achieves R-L 39.61.
- On BigPatent, skimming with pre-sampling removal using GPT-3.5 Turbo reaches BERTScore 82.62 but ROUGE-2 of only 3.31, far below BigBird-Pegasus (42.46).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use keyword-guided segment selection with LDA keywords and sentence-transformer similarity | GPT-3.5 Turbo summarization · GovReport | BERTScore: 86.20 vs 66.0 | Helped |
| Remove redundant segments before sampling rather than after sampling | Document Skimming with GPT-3.5 Turbo · BigPatent | BERTScore: 82.40 -> 82.62 | Helped |
| Remove redundant segments before sampling rather than after sampling | Document Skimming with GPT-3.5 Turbo · BigPatent | R-1: 26.16 -> 27.40 | Mixed |
| Remove redundant segments before sampling rather than after sampling | Document Skimming with GPT-3.5 Turbo · BigPatent | R-2: 2.13 -> 3.31 | Helped |
| Remove redundant segments before sampling rather than after sampling | Document Skimming with GPT-3.5 Turbo · BigPatent | R-L: 20.21 -> 21.25 | Helped |
| Use central truncation instead of skimming with pre-sampling removal | GPT-3.5 Turbo summarization · BigPatent | R-1: 27.40 -> 27.77 | Helped |
| Use central truncation instead of skimming with pre-sampling removal | GPT-3.5 Turbo summarization · BigPatent | R-2: 3.31 -> 3.09 | Hurt |
| Use central truncation instead of skimming with pre-sampling removal | GPT-3.5 Turbo summarization · BigPatent | BERTScore: 82.62 -> 82.57 | No clear effect |
Surprising findings
- Simple central truncation performs competitively with more elaborate segment-selection methods.
- ROUGE-2 scores of the proposed pipelines are far below baselines, while ROUGE-1 and ROUGE-L remain competitive.
Limitations
- ROUGE-2 scores are substantially lower than baselines.
- Some baselines lack BERTScore results due to code unavailability or computational limits.
- Experiments use models with much smaller context sizes (4,096) than the baselines, so the advantage with larger models is untested.
Experimental setup
Datasets: GovReport BigPatent
Benchmarks: GovReport BigPatent
Metrics: ROUGE-1 ROUGE-2 ROUGE-L BERTScore
Baselines: Unlimiformer Hepos PEGASUS-X LLaMA-7B positional interpolation BigBird-Pegasus PRIMERA
Builds on: LDA Unlimiformer LongT5
Closest papers by meaning
- Improving Faithfulness in Abstractive Text Summarization with EDUs Using BART (Student Abstract) (AAAI 2024 · similarity 0.84)
- Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization (ICLR 2026 · similarity 0.84)
- Scaling Up Video Summarization Pretraining with Large Language Models (CVPR 2024 · similarity 0.84)
- BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR 2024 · similarity 0.84)
- Unsupervised Extractive Summarization with Learnable Length Control Strategies (AAAI 2024 · similarity 0.83)
- Document Summarization with Conformal Importance Guarantees (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.