Research map / Large language models / Language, safety and interpretability
Multi-Branch Self-Drafting for LLM Inference Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Draft, a training-free decoding method that uses multi-branch in-context drafting within the LLM's own forward pass, plus an n-gram cache, to speed up autoregressive decoding.
Problem
Autoregressive decoding in LLMs is slow because it generates one token per forward pass; existing draft-and-verify methods need extra draft models, training, or static pre-built caches that mismatch the inference domain.
Approach
Extends autoregressive decoding with multiple parallel draft branches filled with random padding tokens, using custom attention masks so the LLM itself generates contextual draft sequences in the same forward pass; these drafts populate an LRU context cache and are verified in parallel, with no model changes or training.
Key results
- Self-Draft generates 2.0 to 3.2 tokens per forward step across benchmarks, reaching 3.22 on HumanEval for CodeLlama-7B.
- Achieves about 2x end-to-end throughput over autoregressive decoding; e.g., 80% throughput gain on GSM-100 for Llama-7B and 154% on MBPP-100 for CodeLlama-7B.
- Outperforms speculative decoding (Llama-68M draft) and LADE on most model/benchmark pairs, e.g., MT-Bench Llama-7B improves 56% vs LADE 49%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Self-Draft instead of autoregressive decoding | Llama-7B and Llama-13B · GSM-100, MT-Bench; throughput | throughput (tokens/s): 49.0 -> 88.2; 33.5 -> 57.2; 47.1 -> 73.6; 33.6 -> 48.7 | Helped |
| Use Self-Draft instead of autoregressive decoding | CodeLlama-7B and CodeLlama-13B · MBPP-100, HumanEval; throughput | throughput (tokens/s): 42.3 -> 107.3; 32.2 -> 71.5; 47.4 -> 111.2 | Helped |
| Use Self-Draft instead of LADE | Llama-7B and Llama-13B · GSM-100, MT-Bench; throughput | throughput (tokens/s): 82.8 -> 88.2; 53.8 -> 57.2; 70.2 -> 73.6; 45.4 -> 48.7 | Helped |
| Use Self-Draft instead of LADE | CodeLlama-7B and CodeLlama-13B · MBPP-100, HumanEval; throughput | throughput (tokens/s): 98.8 -> 107.3; 67.6 -> 71.5; 95.1 -> 111.2 | Helped |
| Use Self-Draft instead of LADE | CodeLlama-7B · MBPP-100 | decoding efficiency (tokens/forward step): 3.09 -> 2.98 | Hurt |
| Remove drafts from the context cache (ablation) | Self-Draft on Llama-7B and Llama-13B · MT-Bench subtasks | throughput | Hurt |
| Remove drafts from the corpus cache (ablation) | Self-Draft on Llama-7B and Llama-13B · MT-Bench subtasks | throughput | Hurt |
| Increase draft-branch length (ablation) | Self-Draft · MT-Bench, Llama-7B | decoding efficiency | Helped |
| Increase draft-branch length beyond 6 (ablation) | Self-Draft · MT-Bench, Llama-7B | throughput | Hurt |
| Increase the number of draft branches (ablation) | Self-Draft · MT-Bench | decoding efficiency | Helped |
| Increase the number of draft branches when branch length exceeds 7 (ablation) | Self-Draft · MT-Bench | throughput | Hurt |
Surprising findings
- Throughput gains are larger on smaller models (7B vs 13B), attributed to lower drafting overhead relative to model cost.
- Removing the context cache drafts causes a much larger throughput drop than removing corpus cache drafts.
Limitations
- Relies on string matching for candidate retrieval, which fails to capture semantic matches.
- Branch length and count are chosen by grid search, which is inefficient.
- Evaluated only on Llama and CodeLlama models at 7B and 13B sizes with batch size one.
Experimental setup
Datasets: OANC GSM-8K MBPP HumanEval
Benchmarks: MT-Bench GSM-100 MBPP-100 HumanEval
Metrics: throughput (tokens/s) decoding efficiency (tokens per forward step) BLEU ROUGE
Baselines: autoregressive decoding speculative decoding LADE
Builds on: speculative decoding Lookahead decoding Jacobi decoding
Closest papers by meaning
- Adaptive Draft-Verification for Efficient Large Language Model Decoding (AAAI 2025 · similarity 0.93)
- Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration (NeurIPS 2024 · similarity 0.91)
- Tandem Transformers for Inference Efficient LLMs (ICML 2024 · similarity 0.91)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.90)
- AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.90)
- PARD: Accelerating LLM Inference with Low‑Cost PARallel Draft Model Adaptation (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.