Research map / Large language models / Language, safety and interpretability
Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AttnComp, a training-free prompt compression method that scores tokens with query-guided cross-attention from retrieval heads of a small causal LM and groups tokens into semantic units via a maximum spanning tree plus Louvain community detection.
Problem
Prompt compression for long-context LLMs relies on information entropy (PPL) and treats tokens independently, which is suboptimal for deciding which parts of a lengthy prompt to remove.
Approach
Use causal cross-attention from the query to the context (aggregated by max over retrieval heads) as the token importance metric, and use self-attention to cluster tokens into semantic units via a maximum spanning tree and Louvain community detection so units are filtered jointly rather than independently.
Key results
- On NQ with LongChat-13b, AttnComp reaches 68.6 accuracy at 4.5x compression versus 67.1 for LongLLMLingua at 4.5x-equivalent settings (LongLLMLingua 4.6x at 67.1 vs raw 55.3 in efficiency table).
- On LongBench with LongChat-v1.5-7B-32k, AttnComp averages 29.6 versus 25.5 for LongLLMLingua and 24.6 for the original prompt.
- With GPT-3.5-Turbo, AttnComp averages 41.1 on LongBench versus 38.5 for LongLLMLingua and 39.2 for the original prompt; latency drops from 6.31s (raw) to 1.59s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AttnComp instead of LongLLMLingua† | GPT-3.5-Turbo-0613 · NQ (1st position) | accuracy: 75.0 -> 76.6 | Helped |
| Use AttnComp instead of LongLLMLingua† | LongChat-13B-16k · NQ (1st position) | accuracy: 68.7 -> 70.1 | Helped |
| Use AttnComp instead of LongLLMLingua | LongChat-v1.5-7B-32k · LongBench AVG | score: 25.5 -> 29.6 | Helped |
| Use AttnComp instead of LongLLMLingua | GPT-3.5-Turbo-16k · LongBench AVG | score: 38.5 -> 41.1 | Helped |
| Add semantic-unit grouping (ablation) | AttnComp · NQ (LongChat-13B) | accuracy: 67.1 -> 68.6 | Helped |
| Use semantic units instead of phrase units (ablation) | AttnComp · NQ (LongChat-13B) | accuracy: 67.4 -> 68.6 | Helped |
| Use retrieval heads instead of all attention heads (ablation) | AttnComp · NQ (LongChat-13B) | accuracy: 68.2 -> 68.6 | Helped |
| Use cross-attention instead of PPL as the compression metric (ablation) | AttnComp · NQ (LongChat-13B) | accuracy: 61.5 -> 68.6 | Helped |
| Use non-iterative compression instead of iterative token-level compression (ablation) | AttnComp · NQ (LongChat-13B) | accuracy: 67.5 -> 68.6 | Helped |
| Use Mistral-7B-v0.2 instead of Llama-2-7B for fine-grained compression (ablation) | AttnComp · NQ (LongChat-13B) | accuracy: 68.6 -> 68.8 | Helped |
Surprising findings
- Compressed prompts at 2000-token budget on LongBench outperform the uncompressed original prompt for GPT-3.5-Turbo on average (41.1 vs 39.2) in the reported results.
- Attention-based metric retains over 90% of original performance even when the majority of context is filtered out, while PPL degrades significantly.
Limitations
- Compression efficiency decreases for very long prompts, especially during semantic unit identification.
- Small LM context window (Llama-2 4k) requires chunking into 2k windows for long prompts.
- Slight performance drops on Qasper and NarrativeQA for GPT-3.5-Turbo.
Experimental setup
Datasets: Natural Questions (retrieval-augmented setup from Liu et al. 2024) LongBench
Benchmarks: Natural Questions LongBench (Qasper, MultiFieldQA, NarrativeQA, MuSiQue, HotpotQA, 2WikiMultihopQA)
Metrics: accuracy F1 (LongBench metrics) latency compression ratio
Baselines: SBERT OpenAI embedding Cond.PPL Selective-Context LLMLingua LLMLingua-2 LongLLMLingua
Builds on: LLMLingua LongLLMLingua Retrieval heads (Wu et al. 2024) Selective-Context
Closest papers by meaning
- Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference (AAAI 2025 · similarity 0.91)
- Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models (NeurIPS 2024 · similarity 0.90)
- Unveiling and Manipulating Prompt Influence in Large Language Models (ICLR 2024 · similarity 0.90)
- Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks? (ICLR 2025 · similarity 0.90)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.89)
- BatchPrompt: Accomplish more with less (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.