Research map / Architectures and efficiency / Attention and transformers
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MagicDec shows speculative decoding can speed up high-batch, long-context LLM serving by using a self-speculative draft with a sparse (compressed) KV cache, and gives a theoretical model to pick the best drafting strategy.
Problem
Speculative decoding is believed to help only at small batch sizes because verification becomes compute-bound at large batches; the goal is to improve both latency and throughput for long-context serving at large batch sizes without accuracy loss.
Approach
For moderate-to-long sequences, KV cache loading dominates at large batch, so verification cost is amortized and a draft that uses a sparse KV cache (self-speculation) keeps draft cost low and acceptance high; the speedup grows with batch size beyond a critical sequence length. A theoretical speedup formulation selects the draft model size, KV budget, and KV selection algorithm.
Key results
- SnapKV-based self-speculation with Llama-3.1-8B reaches up to 2.51x speedup at batch size 41 with 100K prefill on 8xH100 (CWE task).
- On 8xA100 with Llama-3-8B, speculative decoding achieves speedup once sequence length exceeds about 4000 tokens, and the speedup increases with batch size.
- Speedups of 2.06x for Mistral-7B-v0.3, 1.89x for Qwen-2.5-7B and 1.51x for Qwen-2.5-32B-era settings were reported at large batch sizes on PG-19.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SnapKV instead of StreamingLLM for self-speculation | Llama3.1-8B · PG-19, prefill 32000, batch sizes 16–128 | speedup: 1.08 vs 1.17; 1.26 vs 1.37; 1.55 vs 1.64; 1.85 vs 2.01 | Helped |
| Use a small Llama3.2-1B draft with StreamingLLM KV instead of self-speculation (ablation) | Llama3.1-8B · PG-19, prefill 32000, batch sizes 16–32 | speedup: 1.17 -> 1.27; 1.37 -> 1.46 | Helped |
| Use self-speculation instead of a small draft model with StreamingLLM KV (ablation) | Llama3.1-8B · PG-19, prefill 32000, batch size 128 | speedup: 1.81 -> 2.01 | Helped |
| Increase batch size with long-context speculative decoding | MagicDec, Llama3.1-8B · PG-19, prefill 32000, 8xH100 | speedup: 1.18 -> 1.37 -> 1.62 | Helped |
| Use speculative decoding at prefill length 1024 rather than autoregressive decoding | MagicDec · A100, batch 32, prefill 1024 | speedup: 0.95 | Hurt |
| Increase the draft KV budget for large batch sizes and long sequences (ablation) | MagicDec self-speculation · Large batch, long sequence | speedup | Helped |
| Replace self-speculation with a smaller draft model when context and batch size are modest (ablation) | MagicDec · Shorter sequences and modest batch sizes | speedup | Helped |
| Use SnapKV rather than StreamingLLM for static KV selection | Llama3.1-8B self-speculation · Ruler tasks CWE and QA-1 | theoretical speedup | Helped |
| Use PQCache rather than SnapKV for drafting | Llama3.1-8B self-speculation · Ruler niah-multikeys-3 | theoretical speedup | Helped |
| Use PQCache rather than SnapKV as batch size increases | Llama3.1-8B self-speculation · Ruler tasks at increasing batch size | theoretical speedup | Hurt |
| Use SnapKV instead of a smaller Qwen draft (ablation) | Qwen-2.5-32B · PG-19, prefill 32000, batch sizes 8, 16, 32 | speedup: 1.19 vs 1.02; 1.18 vs 1.16; 1.19 vs 1.34 | Mixed |
Surprising findings
- Speculative decoding speedup increases with batch size for long sequences, contrary to the conventional belief that SD only helps at small batches.
- A KV-compressed self-draft achieves higher token acceptance than a small full-KV draft model (e.g., Llama-3.2-1B) at the same memory budget.
Limitations
- Only decoding is optimized; prefill latency for long contexts is not addressed.
- Gains are larger on high-end GPUs (H100) with higher FLOPS-to-bandwidth ratio; lower-end hardware sees smaller speedups.
- No single KV compression method is recommended; the best choice depends on model, hardware, and task.
Experimental setup
Datasets: PG-19 RULER
Benchmarks: PG-19 RULER (niah-multikeys-3, cwe, qa-1)
Metrics: speedup acceptance rate tokens per step throughput latency per token
Baselines: autoregressive decoding small draft model speculative decoding (Llama3.2-1B) StreamingLLM self-speculation PQCache MLC-LLM speculative decoding
Builds on: speculative decoding TriForce StreamingLLM SnapKV GPT-Fast
Closest papers by meaning
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.93)
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.92)
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.92)
- Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.91)
- Mixture of Attentions For Speculative Decoding (ICLR 2025 · similarity 0.91)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.