Research map / Architectures and efficiency / Attention and transformers
Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Retrofits pre-trained Llama 2 (7B/13B/70B) with Dynamic Memory Compression, a learned per-head online KV-cache merge/append policy, via continued pre-training, achieving up to 4x cache compression without performance loss and up to 7x inference throughput on H100.
Problem
The KV cache of Transformer LLMs grows linearly with sequence length and batch size, making auto-regressive generation memory-bound and inefficient.
Approach
Learn, end-to-end and per head/layer, whether each new token's key-value pair is appended to the cache or accumulated (weighted-averaged) into the last cache entry, using a Gumbel-sigmoid relaxation during training and a global compression-ratio loss; the decisions and importance scores reuse existing q/k neurons so no parameters are added.
Key results
- DMC 4x on Llama 2 7B keeps MMLU at 43.9 vs 44.6 original and CS-QA 70.2 vs 70.5, while GQA 4x drops MMLU to 34.7.
- DMC 4x increases inference throughput by roughly 350-390% for Llama 2 7B and 13B on H100/A100 GPUs, with up to 7x throughput for DMC 8x.
- Retrofitting Llama 2 70B (already GQA 8x) with an extra 2x DMC keeps MMLU at 68.8, for a total 16x cache compression.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DMC instead of up-trained GQA at the same compression ratio | Llama 2 7B/13B · MMLU | accuracy: +5.4 for 2× CR; +9.2 for 4× CR (7B); +4.6 for 2× CR; +5.6 for 4× CR (13B) | Helped |
| Use DMC 2× on top of existing GQA 8× | Llama 2 70B · MMLU, CS-QA, HumanEval | downstream performance | No clear effect |
| Increase DMC compression from 4× to 8× | Llama 2 7B/13B · MMLU | accuracy: -2.8 at 8× CR (7B); -2.4 at 8× CR (13B) | Hurt |
| DMC instead of cache eviction with increasing compression | Llama 2 · HumanEval | Pass@1 | Helped |
| Replace learned dynamic compression with fixed memory pooling (ablation) | DMC · Downstream benchmarks | accuracy | Hurt |
| Use a global compression prior instead of a layer-wise local prior (ablation) | DMC · MMLU | accuracy | Helped |
| Constrain heads in each layer to have similar compression ratios (DMC-C) (ablation) | DMC · MMLU | accuracy: -6.4 in MMLU compared to the ceiling (7B 4×) | Hurt |
| Share decision and importance scores across heads within each layer (DMC-HardC) (ablation) | DMC · MMLU | accuracy | Hurt |
| Use uniform rather than learned importance weights for accumulation (ablation) | DMC-C · Downstream benchmarks | performance | Hurt |
| Set the target compression ratio immediately instead of linearly increasing it (ablation) | DMC · MMLU | accuracy | Hurt |
| Use a shorter versus longer training schedule for increasing compression (ablation) | DMC-C · Downstream performance | performance | Mixed |
| Continue training for a fixed compression ratio after reaching the target (ablation) | DMC · Downstream benchmarks | performance | Helped |
Surprising findings
- Compressing heads in the first few layers reaches extremely high compression for some heads at higher CRs, despite the intuition that early token representations lack context.
- Training DMC from scratch gave negative results versus GQA, so the method works in the retrofitting setting but not (so far) in pre-training from scratch.
Limitations
- Pre-training from scratch with DMC gave negative results compared to GQA in preliminary experiments.
- Throughput gains become sub-linear at high CR as models become compute-bound.
- Training-time implementation uses a window approximation and masks rather than the efficient inference path; H2O/TOVA need materialized attention scores.
Experimental setup
Datasets: The Pile mC4 Common Crawl dumps 2017-2023 ArXiv The Stack PubMed Central Wikipedia
Benchmarks: MMLU HumanEval PIQA BoolQ ARC-C ARC-E HellaSwag WinoGrande
Metrics: MMLU 5-shot accuracy HumanEval pass@1 CS-QA average accuracy inference throughput latency
Baselines: Grouped Query Attention H2O TOVA Original Llama 2
Builds on: Grouped Query Attention Dynamic Token Pooling PagedAttention FlashAttention
Closest papers by meaning
- Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models (ICML 2025 · similarity 0.91)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.90)
- Inference-Time Hyper-Scaling with KV Cache Compression (NeurIPS 2025 · similarity 0.90)
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.90)
- 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.90)
- RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.