Research map / Vision-language and multimodal / Human and embodied modeling
xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes xRAG, which compresses a retrieved document into one token by projecting its dense retrieval embedding into a frozen LLM's space via a trained MLP bridge, using paraphrase pretraining and self-distilled instruction tuning.
Problem
Retrieval-augmented generation inflates LLM inputs with full documents, raising inference cost; existing context compression methods either need large memory for stored activations, require tuning, or achieve low compression rates.
Approach
Treat the document embedding from the dense retriever as a retrieval-modality feature and fuse it into the LLM via a two-layer MLP projector (the only trainable part), trained with paraphrase pretraining and context-aware instruction tuning with KL self-distillation from the uncompressed RAG teacher, so that a single token replaces the document.
Key results
- With Mistral-7b, xRAG averages 42.46 vs 37.49 for TF-IDF and 38.01 for LLMLingua, and 43.63 for uncompressed RAG, using one token instead of 175.1.
- With Mixtral-8x7b, xRAG averages 46.91 (+9.7% over no retrieval), surpassing uncompressed RAG at 46.22 (+8.0%).
- xRAG reduces GFLOPs by an average factor of 3.53 and CUDA time by 1.64x versus uncompressed RAG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace uncompressed retrieved documents with one xRAG token | Mistral-7b RAG · Six-task average | performance: 43.63 -> 42.46 | Hurt |
| replace uncompressed retrieved documents with one xRAG token | Mixtral-8x7b RAG · Six-task average | performance: 46.22 -> 46.91 | Helped |
| remove instruction-tuning finetuning (ablation) | xRAG · Four-task average, Mistral-7b | performance: 44.58 -> 37.87 | Hurt |
| remove paraphrase pretraining (ablation) | xRAG · Four-task average, Mistral-7b | performance: 44.58 -> 38.95 | Hurt |
| add NLL objective (ablation) | xRAG · Four-task average, Mistral-7b | performance: 43.02 -> 44.58 | Helped |
| add self-distillation (ablation) | xRAG · Four-task average, Mistral-7b | performance: 41.49 -> 44.58 | Helped |
| replace full xRAG training with LoRA instruction tuning (ablation) | Mistral-7b · Four-task average | performance: 44.58 -> 39.80 | Hurt |
| remove paraphrase pretraining (ablation) | xRAG · Four-task average, Mixtral-8x7b | performance: 51.45 -> 48.09 | Helped |
| add instruction-tuning finetuning (ablation) | xRAG · Four-task average, Mixtral-8x7b | performance: 40.96 -> 51.45 | Helped |
| add NLL objective (ablation) | xRAG · Four-task average, Mixtral-8x7b | performance: 50.02 -> 51.45 | Helped |
| add self-distillation (ablation) | xRAG · Four-task average, Mixtral-8x7b | performance: 47.94 -> 51.45 | Helped |
| replace the full data blend with reading-comprehension data only (ablation) | xRAG · Four-task average, Mistral-7b | performance: 44.58 -> 44.24 | Hurt |
Surprising findings
- Uncompressed RAG's resilience rate averages only 75.2%, meaning retrieval can flip about a quarter of previously correct answers.
- On Mixtral-8x7b, xRAG slightly outperforms uncompressed RAG on average (46.91 vs 46.22).
Limitations
- Lags behind uncompressed RAG on reasoning-heavy tasks such as HotpotQA and FactKG.
- Only single dense vector retrieval and top-1 document settings were evaluated.
- Training data lacks reasoning-focused examples.
Experimental setup
Datasets: Wikipedia Dec 2021 (37M passages) NQ TriviaQA CoQA DROP NarrativeQA PubMedQA QuAIL SQuAD v2 PwC CNN/DM SamSum
Benchmarks: Natural Questions TriviaQA WebQuestions HotpotQA TruthfulQA FactKG
Metrics: exact match accuracy F1 Rouge-L GFLOPs CUDA time resilience rate boost rate
Baselines: LLMLingua TF-IDF compression RAG without compression no-retrieval LLM
Builds on: LLaVA-style modality bridge Gist LLMLingua FLAN templates
Closest papers by meaning
- VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents (ICLR 2025 · similarity 0.88)
- Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.87)
- Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale (NeurIPS 2026 · similarity 0.87)
- RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations (CVPR 2026 · similarity 0.87)
- RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation (ICLR 2024 · similarity 0.86)
- Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.