Research map / Large language models / Reasoning and chain of thought
LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LaCache, a training-free KV cache method using a ladder-shaped cross-layer retention pattern plus iterative compaction to enable long-range modeling and continuous generation under fixed cache budgets.
Problem
KV cache memory grows linearly with sequence length, causing OOM in long-context generation; existing eviction methods either sacrifice long-range accuracy (recency-based like StreamingLLM) or keep full cache and OOM (retrieval-based like Quest), and importance-based methods need attention maps that
Approach
Stores KV pairs in a ladder-shaped pattern across layers (early tokens kept in shallow layers, later tokens in deeper layers) so more tokens are covered under a fixed budget, and applies this pattern iteratively to already-compacted cache so older tokens are compressed more; attention-free so it stays compatible with FlashAttention.
Key results
- On Wikitext-2 with a 512 cache budget at 1K decoding, LaCache raises Llama2-7B-Chat perplexity only ~5% over full cache (5.20 vs 4.94), versus ~35% for StreamingLLM (6.67).
- On Needle-In-A-Haystack, LaCache lifts Llama3.2-3B-Instruct-128k accuracy from 54.54% to 99.16% at a 50% cache budget.
- On RULER with LongChat-7b-v1.5-32k at 50% cache, LaCache averages 50.88% vs 44.82% for StreamingLLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace StreamingLLM with LaCache at the same 512-token cache budget | Llama2-7B-Chat · Wikitext-2-raw-v1, 1K decoding | perplexity: 6.67 -> 5.20 | Helped |
| use LaCache instead of StreamingLLM at 50% cache | Llama2-7B-Chat · LongBench average | score: 26.56 -> 27.34 | Helped |
| use LaCache instead of StreamingLLM at 50% cache | Llama2-13B-Chat · LongBench average | score: 28.30 -> 29.22 | Helped |
| use LaCache instead of StreamingLLM at 50% cache | SmolLM2-1.7B-Instruct · LongBench average | score: 22.59 -> 23.05 | Helped |
| use LaCache instead of StreamingLLM at 50% cache | LongChat-7b-v1.5-32k · RULER average | accuracy: 44.82 -> 50.88 | Helped |
| use LaCache instead of StreamingLLM at 50% cache | Llama3.2-3B-Instruct-128k · Needle-In-A-Haystack | accuracy: 54.54% -> 99.16% | Helped |
| use LaCache instead of StreamingLLM at 25% cache | LongChat-7b-v1.5-32k · Needle-In-A-Haystack | accuracy: 33.40% -> 65.30% | Helped |
| increase overlap O from 0 to S/2 (ablation) | LaCache · Long-context understanding, QA tasks | score: 19.48 -> 18.48 | Hurt |
| increase overlap O from 0 to S/2 (ablation) | LaCache · Long-context understanding, synthetic tasks | score: 5.17 -> 6.17 | Helped |
| increase overlap O from 0 to S/4 (ablation) | LaCache · Long-context understanding, QA tasks | score: 19.48 -> 18.94 | Hurt |
| increase overlap O from 0 to S/4 (ablation) | LaCache · Long-context understanding, synthetic tasks | score: 5.17 -> 5.67 | Helped |
Surprising findings
- LaCache outperforms full cache perplexity on some settings where full cache explodes beyond pretraining length (e.g., Llama3-8B at 16K decoding: 109.94 full vs 5.36-5.19 range for compressed methods).
Limitations
- Ladder pattern may not be optimal for every scenario; alternative patterns could improve memory efficiency and performance.
- Method is purely training-free; fine-tuning to adapt the KV pattern to tasks is not explored.
- Mixed results on some LongBench tasks, e.g., larger overlap hurts QA tasks while helping synthetic tasks.
Experimental setup
Datasets: Wikitext-2 PG19 LongBench Needle-In-A-Haystack RULER
Benchmarks: Wikitext-2 PG19 LongBench Needle-In-A-Haystack RULER
Metrics: perplexity accuracy score-throughput trade-off
Baselines: StreamingLLM H2O TOVA PyramidInfer SnapKV Quest
Builds on: StreamingLLM FlashAttention
Closest papers by meaning
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.93)
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.93)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.92)
- SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.92)
- $\text{D}_{2}\text{O}$: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models (ICLR 2025 · similarity 0.91)
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.