Research map / Architectures and efficiency / Attention and transformers
Inference-Time Hyper-Scaling with KV Cache Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Dynamic Memory Sparsification (DMS), a cheaply retrofitted learned KV-cache eviction method with delayed eviction that enables inference-time hyper-scaling (more tokens/threads at equal memory reads or latency).
Problem
Inference-time scaling of reasoning LLMs is bottlenecked by KV cache size and memory reads; training-free KV eviction degrades accuracy at high compression, while learned compression (DMC) is expensive to retrofit.
Approach
Retrofit pretrained LLMs with a learned per-head binary eviction decision (Gumbel-sigmoid) whose eviction is executed after a delay window, distilled from the original model's logits, so that KV cache can be compressed 8x with about 1K training steps while keeping accuracy and enabling more reasoning tokens or threads for the same budget.
Key results
- DMS improves Qwen-R1 32B by 12.0 points on AIME 24, 8.6 on GPQA Diamond and 9.7 on LiveCodeBench on average at equal KV-cache memory reads (Pareto frontier difference).
- DMS 8x on Qwen3-8B is within about 1-2 points of vanilla on most general tasks (e.g., AIME 2024 73.0 vs 75.0, GPQA 57.6 vs 58.8) and gives up to 5x higher throughput on LiveCodeBench.
- On Llama 3.2 1B at 4x compression, DMS scores 42.3 on GSM8K versus 47.0 vanilla, and 96.8 on NIAH versus 96.4, outperforming training-free baselines and DMC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DMS 8× instead of vanilla KV-cache retention | Qwen3-8B · MATH-500 | accuracy: 95.1 -> 95.5 | Helped |
| Use DMS 8× instead of vanilla KV-cache retention | Qwen3-8B · HumanEval | accuracy: 87.8 -> 89.6 | Helped |
| Use DMS 8× instead of vanilla KV-cache retention | Qwen3-8B · IFEval | accuracy: 90.3 -> 88.8 | Hurt |
| Use DMS 8× instead of vanilla KV-cache retention | Qwen3-8B · GPQA Diamond | accuracy: 58.8 -> 57.6 | Hurt |
| Use DMS 8× instead of vanilla KV-cache retention | Qwen3-8B · AIME 2024 | accuracy: 75.0 -> 73.0 | Hurt |
| Use DMS 8× instead of vanilla KV-cache retention | Qwen3-8B · ArenaHard v0.1 | accuracy: 88.4 -> 89.7 | Helped |
| Increase DMS compression from 2× to 4× (ablation) | Llama 3.2 1B Instruct · GSM8K | accuracy: 46.9 -> 42.3 | Hurt |
| Add a 16-token delayed-eviction window rather than immediate eviction (ablation) | DMS · GSM8K 0-shot | accuracy | Helped |
| Increase sliding-window size (ablation) | DMS · reasoning tasks | accuracy | Helped |
| Use DMS instead of vanilla at CR2 with a 16-token delay window (ablation) | Llama 3.2 1B Instruct · NIAH, 4K context | accuracy: 96.4 -> 97.8 | Helped |
| Use DMS instead of vanilla at CR2 with a 16-token delay window (ablation) | Llama 3.2 1B Instruct · Variable Tracking, 4K context | accuracy: 55.8 -> 63.2 | Helped |
| Retrofitting with DMS rather than DMC using fewer training tokens (ablation) | DMS retrofitting · GSM8K 0-shot | accuracy: 8× fewer training tokens | Helped |
Surprising findings
- On long-context Variable Tracking, DMS variants surpass the vanilla model, and on NIAH the compressed model sometimes exceeds vanilla.
- Immediate eviction degrades accuracy sharply at every window size, while a short 16-token delayed-eviction window retains reasoning at 4x.
Limitations
- Evaluated only up to 32B parameters, 32K context and up to 8x compression.
- Not combined with alternative attention variants such as MLA or with query-selection methods like Quest.
- Compression ratio drifts with sequence length, sparsifying less early and more aggressively beyond about 10K tokens.
Experimental setup
Datasets: OpenR1-Math-220k DeepMind mathematics dataset (1D linear algebra) curated programming corpus
Benchmarks: AIME 2024 MATH-500 GPQA Diamond LiveCodeBench GSM8K MMLU HellaSwag Needle in a Haystack Variable Tracking MMLU-Pro HumanEval IFEval
Metrics: exact match pass@all KV-cache token reads peak tokens in memory throughput Pareto frontier difference
Baselines: Vanilla LLM DMC TOVA H2O Quest
Builds on: Dynamic Memory Compression Quest TOVA Heavy-Hitter Oracle Megatron-LM
Closest papers by meaning
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.91)
- Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference (ICML 2024 · similarity 0.90)
- Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference (ICML 2024 · similarity 0.90)
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.90)
- KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation (ICML 2024 · similarity 0.90)
- MUSTAFAR: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.