Research map / Large language models / Language, safety and interpretability
Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CPC, a sentence-level prompt compressor that ranks context sentences by a contrastively trained context-aware sentence encoder (Mistral-7B with LoRA), trained on a newly generated CQR dataset, and reports better accuracy and up to 10.93x faster inference than LongLLMLingua.
Problem
Long prompts raise LLM inference cost, but token-level prompt compression can break sentence coherence and semantics at high compression ratios and is computationally expensive.
Approach
Compress prompts at the sentence level by scoring each sentence's relevance to the question with a context-aware sentence encoder (token embeddings pooled over the sentence span within its full context), trained contrastively on questions with positive and negative context sentences generated and filtered with LLMs.
Key results
- On LongBench at 2,000 tokens, CPC averages 49.5 versus 48.0 for LongLLMLingua (SOTA), and at 3,000 tokens 50.0 versus 48.8.
- CPC is up to 10.93x faster than LongLLMLingua at median relative latency (1x for CPC vs 10.93x for LongLLMLingua in Table 3 labeling).
- Replacing the encoder with MpNet-v2 or LLM2Vec lowers LongBench average by 2.3 and 6.6 points respectively, per the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CPC instead of LongLLMLingua | prompt compression · LongBench, 3,000 tokens | AVG: 48.8 -> 50.0 | Helped |
| use CPC instead of LongLLMLingua | prompt compression · ZeroSCROLLS, 3,000 tokens | AVG: 32.8 -> 34.9 | Helped |
| use CPC instead of LongLLMLingua | prompt compression · LongBench, 2,000 tokens | AVG: 48.0 -> 49.5 | Helped |
| use CPC instead of LongLLMLingua | prompt compression · ZeroSCROLLS, 2,000 tokens | AVG: 32.5 -> 33.8 | Helped |
| use CPC instead of LongLLMLingua | prompt compression · LongBench | average relative latency: 27.5× | Helped |
| add L_MNTP to the training loss (ablation) | CPC context-aware sentence encoder · LongBench, SingleDoc, 2,000 tokens | accuracy: 35.46 -> 42.38 | Helped |
| add L_SC to the training loss (ablation) | CPC context-aware sentence encoder · LongBench, SingleDoc, 2,000 tokens | accuracy: 41.17 -> 42.38 | Helped |
| add question verification during dataset collection (ablation) | CPC · LongBench, SingleDoc, 2,000 tokens | accuracy: 41.51 -> 42.38 | Helped |
| add similarity-based negative filtering (ablation) | CPC · LongBench, SingleDoc, 2,000 tokens | accuracy: 41.34 -> 42.38 | Helped |
| add KL-based negative filtering (ablation) | CPC · LongBench, SingleDoc, 2,000 tokens | accuracy: 40.71 -> 42.38 | Helped |
| use 4 negatives per positive instead of 2 (ablation) | CPC context-aware sentence encoder · LongBench, SingleDoc, 2,000 tokens | accuracy: 42.38 -> 41.52 | Hurt |
| use 8 negatives per positive instead of 2 (ablation) | CPC context-aware sentence encoder · LongBench, SingleDoc, 2,000 tokens | accuracy: 42.38 -> 41.06 | Hurt |
Surprising findings
- LongLLMLingua has high latency on some short-context samples due to dependence on text structure, unlike the roughly linear scaling of CPC.
Limitations
- LLMLingua-2 is faster than CPC at inference, so CPC does not win on latency across all methods.
- Training data depends on proprietary LLMs (gpt-3.5-turbo-instruct) and filtering thresholds chosen by the authors.
- Results on a larger GPT-4o backbone are reported only in aggregate in the text.
Experimental setup
Datasets: WikiText CQR dataset LongBench ZeroSCROLLS PubMed Krapivin-2009 MeetingBank SummScreen
Benchmarks: LongBench ZeroSCROLLS PubMed Krapivin-2009 MeetingBank SummScreen
Metrics: F1 ROUGE Levenshtein similarity keyword recall latency
Baselines: LLMLingua LLMLingua-2 LongLLMLingua Selective-Context BM25 SBERT OpenAI embeddings MpNet-v2 LLM2Vec
Builds on: LLMLingua LongLLMLingua LLMLingua-2 Sentence-BERT LLM2Vec
Closest papers by meaning
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.91)
- Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models (NeurIPS 2024 · similarity 0.90)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.90)
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.89)
- RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation (ICLR 2024 · similarity 0.89)
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.