Research map / Architectures and efficiency / Attention and transformers
SCBench: A KV Cache-Centric Analysis of Long-Context Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Existing long-context benchmarks evaluate efficient methods in single-request settings and ignore the full KV cache lifecycle, including reuse across turns and requests.
Approach
Organizes long-context efficiency methods into four KV-cache stages (generation, compression, retrieval, loading) and builds a benchmark of shared-context multi-turn and multi-request sessions to expose failures of query-conditioned sparse/compressed KV caches; also proposes Tri-shape, a training-free sparse prefill pattern.
Key results
- Sparse encoding with dense decoding (A-shape, Tri-shape, MInference) retains strong multi-request performance, while KV cache dropping methods like StreamingLLM and SnapKV drop sharply after the first turn (e.g., SnapKV multi-request avg 15.2 vs 37.2 full on Llama-3.1-8B).
- Tri-shape, a training-free sparse attention method, improves first-turn results over A-shape, reaching 30.3 multi-turn avg vs 27.1 for A-shape on Llama-3.1-8B.
- Dynamic sparse attention MInference is the strongest efficient method, with multi-turn avg 42.5 on Llama-3.1-8B against 48.7 for full attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dense bottom query rows to A-shape sparse attention (Tri-shape) | A-shape · SCBench first-turn performance and multi-request mode | accuracy | Helped |
| Add dense bottom query rows to A-shape sparse attention (Tri-shape) | Llama-3.1-8B · SCBench multi-turn mode, AVG. | AVG.: 27.1 -> 30.3 | Helped |
| Add dense bottom query rows to A-shape sparse attention (Tri-shape) | Llama-3.1-8B · SCBench multi-request mode, AVG. | AVG.: 27.6 -> 25.9 | Hurt |
| Use sparse encoding with dense decoding rather than sparse decoding | long-context methods · SCBench shared-context multi-turn and multi-request scenarios | accuracy | Helped |
| Use sub-O(n) KV cache methods instead of O(n) memory methods | long-context methods · SCBench multi-turn follow-up requests | accuracy | Hurt |
| Use prompt compression (LLMLingua-2) | long-context LLMs · Global information tasks, including many-shot ICL | performance | Helped |
| Use prompt compression (LLMLingua-2) | long-context LLMs · Retrieval tasks | performance | Hurt |
| Remove query access (ablation) | Tri-shape · SCBench multi-turn mode | Retr.Semantic: 31.4~/~25.7 | Hurt |
| Remove query access (ablation) | MInference · SCBench multi-turn mode | Retr.Semantic: 40.4~/~35.6 | Hurt |
| Remove query access (ablation) | SnapKV · SCBench multi-turn mode | Retr.Semantic: 19.0~/~9.7 | Hurt |
| Use MInference instead of static sparse attention | long-context LLMs · SCBench multi-turn mode | AVG. | Helped |
| Use KV cache dropping (StreamingLLM or SnapKV) | long-context LLMs · SCBench shared-context retrieval and multi-turn tasks | accuracy | Hurt |
Surprising findings
- Sparse encoding with dense decoding performs robustly across requests, while sparse decoding degrades sharply in multi-turn settings.
- Hybrid Jamba-1.5-Mini scores higher than many Transformer compression methods in multi-request mode (48.0 vs 37.2 full attention on Llama-3.1-8B, though not the same model).
Limitations
- Multi-turn follow-up turns use ground-truth answers rather than model-generated content.
- Results are reported for only a subset of methods on each base model.
Experimental setup
Datasets: InfiniteBench RepoQA ManyShotICL Big-Bench Hard RULER arXiv papers
Benchmarks: SCBench InfiniteBench LongBench
Metrics: accuracy ROUGE Pass@1
Baselines: A-shape MInference StreamingLLM SnapKV PyramidKV KIVI CacheBlend Quest RetrievalAttention LLMLingua-2 Codestral Mamba Jamba-1.5-Mini
Builds on: InfiniteBench RULER RepoQA MInference SnapKV
Closest papers by meaning
- ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference (NeurIPS 2025 · similarity 0.94)
- Training Free Exponential Context Extension via Cascading KV Cache (ICLR 2025 · similarity 0.93)
- LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models (ICML 2025 · similarity 0.93)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.93)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.92)
- MiniCache: KV Cache Compression in Depth Dimension for Large Language Models (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.