Research map / Architectures and efficiency / Attention and transformers
Training Free Exponential Context Extension via Cascading KV Cache
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free cascading KV cache, built from sub-caches with decreasing acceptance rates and EMA-based token selection, plus a strided linear prefill, to extend the effective context of sliding-window LLM inference.
Problem
Sliding-window and sink KV-cache methods for LLMs evict tokens too early and have slow linear prefill, so they lose long-range information and can be slower than quadratic flash attention on realistic lengths.
Approach
Organize a fixed-size KV cache into cascading sub-caches that accept progressively fewer evicted tokens, and retain tokens with higher attention-score EMA when evicting, extending the token span without more memory; combined with strided chunked prefill for linear prompt processing.
Key results
- Cascading KV cache gives a 12.13% average improvement on LongBench over the next best linear model and a 4.48% average ROUGE gain on BookSum among linear baselines.
- Strided prefill reduces 1M-token attention latency by a factor of 6.8 relative to FlashAttention-2.
- On passkey retrieval with a 65K cache, the method stays above random (10%) accuracy after four doublings of context beyond the cache size, while StreamingLLM drops to near random after the first doubling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cascading KV cache instead of Streaming LLM | Qwen2-7B · PG19 test, 16K cache | perplexity: 9.18 -> 9.14 | Helped |
| Use cascading KV cache instead of Streaming LLM | LLaMA3.1-8B · PG19 test, 32K cache | perplexity: 7.65 -> 7.60 | Helped |
| Use cascading KV cache instead of Streaming LLM | linear inference methods · Booksum | Rouge-L: 19.85 -> 20.52 | Helped |
| Use cascading KV cache instead of Streaming LLM | linear inference methods · LongBench, mean | score: 26.37 -> 31.8 | Helped |
| Use cascading KV cache instead of Streaming LLM | LLaMA3.1-8B · MMLU, overall | accuracy: 66.98 -> 67.11 | Helped |
| Use cascading KV cache with strided prefill | Flash Attention 2 · 1M tokens | attention latency: 14.8\% of the time | Helped |
| Add attention-score token selection (ablation) | Cascading KV Cache · PG19, first book, cache size 2048 | perplexity: 8.03 -> 7.88 | Helped |
| Increase strided prefill size (ablation) | Cascading KV Cache · PG19, first book | perplexity | Helped |
| Increase the number of cascades up to 8 (ablation) | Cascading KV Cache · Passkey retrieval | accuracy | Helped |
| Increase the number of cascades beyond 8 (ablation) | Cascading KV Cache · Passkey retrieval | accuracy | Hurt |
| Use independent rather than homogeneous attention heads (ablation) | Cascading KV Cache · PG19 | perplexity | Helped |
| Use median rather than mean or max head reduction (ablation) | Cascading KV Cache with independent heads · PG19 | perplexity | Hurt |
Surprising findings
- Token selection alone (without cascades) makes no perplexity difference over StreamingLLM, showing the gain depends on selecting high-attention tokens.
- Accuracy improves with more cascades only up to about 8 cascades; beyond that, sparsity hurts performance.
Limitations
- Tokens must still be discarded to keep linear complexity, so extrapolation is bounded; performance drops beyond about 8 cascades.
- Evaluated only on Llama 3.1 and Qwen2 models and a limited set of benchmarks.
- Sparsity from cascades trades off against accuracy; token span predicts accuracy only up to about 4 cascades.
Experimental setup
Datasets: PG19 LongBench BooksSum InfiniteBench MMLU
Benchmarks: PG19 LongBench BooksSum passkey retrieval MMLU InfiniteBench
Metrics: perplexity accuracy ROUGE latency
Baselines: StreamingLLM BigBird FlashAttention-2 H2O SnapKV MInference PyramidKV Vanilla (truncated)
Builds on: StreamingLLM FlashAttention-2 H2O SnapKV
Closest papers by meaning
- SCBench: A KV Cache-Centric Analysis of Long-Context Methods (ICLR 2025 · similarity 0.93)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.92)
- KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse (NeurIPS 2025 · similarity 0.91)
- LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models (ICML 2025 · similarity 0.91)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.90)
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.