Research map / Large language models / Retrieval and knowledge
RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RECOMP, which compresses retrieved documents into short textual summaries (extractive or abstractive, with empty output for selective augmentation) before prepending them to frozen black-box LMs.
Problem
Retrieval-augmented LMs prepend long retrieved documents, which raises inference cost, makes LMs miss information, and lets irrelevant context confuse them.
Approach
Insert a trained compressor between retriever and frozen LM: an extractive dual-encoder selects useful sentences via contrastive learning on LM feedback, and an abstractive T5 is symbolically distilled from GPT-3.5 summaries filtered by end-task performance, with empty summaries enabling selective augmentation.
Key results
- On WikiText-103 with GPT-2, the abstractive compressor reaches PPL 33.64 using 15 tokens versus 35.53 for prepending top 5 documents (512 tokens).
- On NQ with Flan-UL2 (20B), the abstractive compressor achieves EM 37.04 with 36 tokens versus 39.39 EM for top 5 documents (660 tokens).
- On HotpotQA, the extractive compressor reaches EM 30.40 with 75 tokens, outperforming top 5 documents at 32.80 EM only partially; it beats BM25, DPR and Contriever baselines on EM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace one uncompressed document with the trained abstractive compressor | GPT-2 on WikiText-103 · WikiText-103 | PPL: 32.90 -> 33.64 | No clear effect |
| Use trained abstractive compression instead of off-the-shelf T5 | GPT-2 on WikiText-103 · WikiText-103 | PPL: 37.80 -> 33.64 | Helped |
| Use trained extractive compression instead of Contriever sentence ranking | Language modeling compressors · WikiText-103 | PPL: 35.54 -> 33.67; 18.98 -> 18.19; 11.05 -> 10.73 | Helped |
| Prepend top 5 documents instead of top 1 document | GPT-2 · WikiText-103 | PPL: 32.90 -> 35.53 | Hurt |
| Replace top 5 uncompressed documents with the trained abstractive compressor | Flan-UL2 · NQ dev | EM: 39.39 -> 37.04 | Hurt |
| Replace top 5 uncompressed documents with the trained abstractive compressor | Flan-UL2 · TriviaQA test | EM: 62.37 -> 58.68 | Hurt |
| Replace top 5 uncompressed documents with the trained extractive compressor | Flan-UL2 · HotpotQA dev sample | EM: 32.80 -> 30.40 | Hurt |
| Use trained abstractive compression instead of trained extractive compression | Flan-UL2 · NQ and TriviaQA | EM: 36.57 -> 37.04; 58.99 -> 58.68 | Mixed |
| Use trained extractive compression instead of trained abstractive compression | Flan-UL2 · HotpotQA dev sample | EM: 28.20 -> 30.40 | Helped |
| Use selective augmentation instead of always prepending GPT-3.5 summaries | LLaMA-13B · NQ | EM: 35.71 -> 45.60 | Helped |
| Use selective augmentation instead of always prepending GPT-3.5 summaries | LLaMA-13B · TriviaQA | EM: 67.89 -> 74.37 | Helped |
| Use selective augmentation instead of always prepending GPT-3.5 summaries | LLaMA-13B · HotpotQA | EM: 34.60 -> 41.40 | Helped |
Surprising findings
- Off-the-shelf GPT-3.5 summaries are not reliably better than trained compressors: manual analysis finds the trained abstractive compressor less faithful than GPT-3.5 but more comprehensive on NQ and TQA.
- Prepending top 1 document outperforms prepending top 5 documents for language modeling, indicating irrelevant documents can hurt performance.
Limitations
- Abstractive compressor is less faithful than GPT-3.5 and struggles with multi-document synthesis on HotpotQA.
- Extractive compressor does not perform selective augmentation; it prepends a fixed number of sentences.
- Transfer to LLaMA-13B is worse than the base-model setting and leaves a gap to oracle compression.
Experimental setup
Datasets: WikiText-103 Natural Questions TriviaQA HotpotQA Wikipedia dump Dec. 20, 2018
Benchmarks: WikiText-103 Natural Questions TriviaQA HotpotQA
Metrics: perplexity exact match F1 number of tokens prepended faithfulness (manual) comprehensiveness (manual)
Baselines: BoW Named Entities BM25 Contriever DPR Random T5 (large, 770M) GPT-3.5-turbo
Builds on: Contriever symbolic knowledge distillation T5 DPR Flan-UL2
Closest papers by meaning
- Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference (AAAI 2025 · similarity 0.89)
- Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.89)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.89)
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.88)
- ARC-Encoder: learning compressed text representations for large language models (NeurIPS 2026 · similarity 0.88)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.