Research map / Architectures and efficiency / RNNs and normalization
Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LESS, which pairs an eviction-based sparse KV cache with a constant-sized low-rank recurrent state that absorbs evicted KV pairs, trained per layer with small kernel MLPs to approximate the residual attention.
Problem
KV cache memory grows linearly with sequence length in LLM decoding, and sparse eviction policies discard tokens irrecoverably, degrading performance on tasks needing recall of many earlier tokens.
Approach
Accumulate evicted KV pairs into a constant-size low-rank recurrent state (learned kernels phi/psi, updated recursively) that is added to the sparse attention output, so every previous token contributes; each layer's kernels are trained independently with frozen base weights to match the full attention output.
Key results
- On WikiText with 2% H2O in Llama 2 7B, LESS reduces word perplexity to 10.745 vs 13.333 for the sparse-only baseline (full cache 8.791).
- On Falcon 7B CNN/DailyMail at 10% H2O, LESS (10%) reaches Rouge-1 23.22 vs 21.26 for the baseline (full cache 25.92).
- LESS adds a constant state equivalent to about 4 extra KV tokens (R=8) and reports 1.1-1.3x latency reduction vs full cache and 1.7x throughput gain for Llama 2 7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the learned low-rank state to 5% H2O | Llama 2 7B · WikiText | word perplexity: 9.863 -> 9.657 | Helped |
| Add the learned low-rank state to 5% H2O | Llama 2 7B · PG-19 | word perplexity: 27.939 -> 27.089 | Helped |
| Add the learned low-rank state to 5% Lambda-masking | Llama 2 7B · WikiText | word perplexity: 10.66 -> 10.12 | Helped |
| Add the learned low-rank state to 5% Lambda-masking | Llama 2 7B · MuTual | R@1: 53.50 -> 54.51 | Helped |
| Add the learned low-rank state to 5% Lambda-masking | Llama 2 7B · BoolQ | accuracy: 77.28 -> 78.65 | Helped |
| Add the learned low-rank state to H2O | Llama 2 13B · CNN/DailyMail | Rouge-1: 23.57 -> 25.27 | Helped |
| Add the learned low-rank state to H2O | Llama 2 13B · XSum | Rouge-1: 33.09 -> 33.40 | Mixed |
| Add the learned low-rank state to H2O | Llama 2 7B · MultiNews | Rouge-1: 13.38 -> 15.42 | Helped |
| Add the learned low-rank state to H2O | Llama 2 7B · CNN/DailyMail | Rouge-1: 18.18 -> 19.21 | Helped |
| Add the learned low-rank state to H2O | Llama 2 7B · XSum | Rouge-1: 29.03 -> 30.72 | Mixed |
| Use a learned low-rank state rather than Performer kernels | H2O + Performer, Llama 2 7B · WikiText, 5% H2O | word perplexity: 9.863 -> 9.657 | Helped |
| Add the low-rank state to full-cache-sized sparse decoding | Llama 2 7B · generation, 2048+2048, batch 24 | latency (s): 78.3 -> 95.1 | Hurt |
Surprising findings
- Plain H2O with Performer kernels gives little improvement over the sparse baseline, suggesting learned kernels are necessary.
- Lambda-masking with LESS still yields very high perplexity on PG-19 for the baseline (22K), though LESS reduces it to 3.9K.
Limitations
- LESS remains far from full-cache performance in many settings, especially at very low sparsity levels.
- Latency is slightly higher than the sparse-only H2O baseline due to low-rank state operations.
Experimental setup
Datasets: C4
Benchmarks: WikiText PG-19 MuTual BoolQ CNN/DailyMail XSum MultiNews
Metrics: word perplexity Rouge-1/2/L accuracy recall@1 latency throughput Hellinger distance
Baselines: Baseline (sparse-only) Baseline+ (sparse with extra tokens) H2O+Performer TOVA Full Cache
Builds on: H2O (heavy-hitter oracle) Performer Λ-masking (StreamingLLM) TOVA linear transformers
Closest papers by meaning
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.93)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.93)
- Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction (NeurIPS 2026 · similarity 0.93)
- SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs (ICML 2025 · similarity 0.93)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.92)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.