Research map / Large language models / Language, safety and interpretability
Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EHPC, a training-free prompt compression method that uses attention scores from identified "evaluator heads" in the first few layers during pre-filling to drop unimportant tokens, evaluated against LLMLingua-family methods and KV-cache compression baselines.
Problem
Long-context inputs raise LLM inference cost and latency and can hurt performance; existing prompt compression methods are either costly (LongLLMLingua's repeated LLM calls) or require training a secondary compressor (LLMLingua-2).
Approach
Identify a small set of attention heads (evaluator heads) in one layer whose attention to evidence in needle-in-a-haystack synthetic data is high, then use their pooled attention scores to score and delete tokens, running only the first few layers during pre-filling so the compressed prompt can be sent to the same or a different model.
Key results
- Under a 2,000-token constraint on LongBench, EHPC (EMI) averages 49.6 versus 48.0 for LongLLMLingua and 39.1 for LLMLingua-2.
- On LongBench latency subset, EHPC compression takes 0.88 s versus 67.44 s for LongLLMLingua and 1.27 s for LLMLingua-2, with total latency 1.99 s.
- With Phi-3.5-mini at 2048 KV budget, EHPC reaches an average of 36.46 versus 34.20 for SnapKV-2048, and the paper reports QA gains of up to 40% over direct inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EHPC (EMI) instead of LongLLMLingua at a 2,000-token prompt constraint | LongBench prompt compression · LongBench, 2,000-token constraint | average score: 48.0 -> 49.6 | Helped |
| Use EHPC (EMI) instead of LLMLingua-2 at a 2,000-token prompt constraint | ZeroSCROLLS prompt compression · ZeroSCROLLS, 2,000-token constraint | average score: 33.4 -> 34.6 | Helped |
| Use EHPC (EMI) instead of LongLLMLingua at a 3,000-token prompt constraint | ZeroSCROLLS prompt compression · ZeroSCROLLS, 3,000-token constraint | average score: 32.8 -> 36.7 | Helped |
| Use EHPC compression instead of direct inference on the full prompt | LongBench prompt compression latency subset · 2,048-token compression latency subset | total latency (seconds): 2.16 -> 1.99 | Helped |
| Use EHPC-2048 instead of GemFilter-2048 | LLaMA 3.1 8B Instruct · LongBench, 2,048-token KV cache budget | average score: 35.87 -> 37.86 | Helped |
| Use EHPC-1024 instead of SnapKV-1024 | Phi 3.5 Mini 3.8B Instruct · LongBench, 1,024-token KV cache budget | average score: 33.68 -> 39.22 | Helped |
| Use EHPC-2048 instead of SnapKV-2048 | Phi 3.5 Mini 3.8B Instruct · LongBench, 2,048-token KV cache budget | average score: 34.20 -> 36.46 | Helped |
| Select evaluator heads rather than comparison heads from the same layer | LongBench MF-en · MF-en | score: 24.3 -> 27.7 | Helped |
| Identify heads with task-customized multi-hop probe data instead of QA probe data | LongBench · Multi-hop | score: 16.40 -> 16.87 | No clear effect |
| Identify heads with task-customized code probe data instead of QA probe data | LongBench · Code | score: 47.86 -> 47.72 | No clear effect |
| Use EHPC prompt compression rather than KV-cache compression | LongBench acceleration · Code and few-shot learning tasks | performance | Hurt |
| Use EHPC prompt compression rather than KV-cache compression | LongBench acceleration · Question-answering tasks | performance | Helped |
Surprising findings
- Prompt compression with EHPC on Phi-3.5-mini surpasses full-context (All KV) performance on several QA tasks, which the authors attribute to reduced noisy context.
- Heads identified with customized task-aware probe data do not significantly outperform heads identified from simple QA needle data, suggesting task-agnostic evaluator heads.
Limitations
- Relies on a needle-in-a-haystack pilot to pick heads, and results depend on the chosen model and hyperparameters (observation window, kernel size) per model.
- Code and task-related tasks and few-shot learning show performance declines relative to KV cache methods.
- Complexity analysis is theoretical and assumes compression ratios of at least 2; the paper lacks broader model coverage beyond three LLMs.
Experimental setup
Datasets: Needle-in-a-Haystack Ruler multi-hop tracing LongBench ZeroSCROLLS
Benchmarks: LongBench ZeroSCROLLS LongBench latency subset (RepoBench-P)
Metrics: F1 Rouge-L accuracy exact match edit similarity latency (seconds)
Baselines: BM25 SBERT OpenAI embedding Selective-Context LLMLingua LLMLingua-2 LongLLMLingua H2O SnapKV GemFilter
Builds on: LLMLingua LongLLMLingua retrieval heads SnapKV GemFilter
Closest papers by meaning
- Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · similarity 0.90)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.90)
- Long Context Compression with Activation Beacon (ICLR 2025 · similarity 0.90)
- Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models (NeurIPS 2024 · similarity 0.89)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.89)
- Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.