Research map / Large language models / Reasoning and chain of thought
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SWIFT, a plug-and-play self-speculative decoding method that uses on-the-fly layer skipping, with context-based layer-set optimization and confidence-aware drafting, achieving 1.3-1.6x lossless speedups without auxiliary models or training.
Problem
Speculative decoding usually needs extra draft-model parameters or extensive training, and prior layer-skipping self-speculative methods (Self-SD) need hours of offline Bayesian optimization and are not task-generalizable; Jacobi-based plug-and-play methods give limited speedups.
Approach
Use the target LLM with a subset of intermediate layers skipped as the draft model, and adaptively optimize the skipped-layer set during inference using a step-level matchness score on LLM-generated context (random search plus periodic Bayesian optimization), combined with early-stopping drafting and confidence-based dynamic top-k verification.
Key results
- SWIFT achieves 1.3x-1.6x wall-clock speedup over vanilla decoding across LLaMA-2 models (e.g., 1.41x overall on LLaMA-2-13B greedy, 1.48x on LLaMA-2-70B) with acceptance rates of 98%-100% on LLaMA-2 series.
- On LLaMA-2-13B CNN/DM, SWIFT matches Self-SD speedup (1.56x) while cutting optimization latency from ~7.5 hours to ~2 minutes (116s).
- Compared with Lookahead Decoding (1.27x overall on LLaMA-2-13B), SWIFT reaches 1.41x; Parallel Decoding gives 0.97x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use on-the-fly layer-skipping speculative decoding instead of Lookahead | LLaMA-2-13B · CNN/DM | speedup: 1.16× -> 1.37× | Helped |
| use on-the-fly layer-skipping speculative decoding instead of Lookahead | LLaMA-2-13B · GSM8K | speedup: 1.29× -> 1.31× | Helped |
| use on-the-fly layer-skipping speculative decoding instead of Lookahead | LLaMA-2-13B · TinyStories | speedup: 1.37× -> 1.53× | Helped |
| add early-stopping drafting (ablation) | Swift · CNN/DM | speedup: 0.896× -> 1.560× | Helped |
| add early-stopping drafting (ablation) | Swift · CNN/DM | token acceptance rate: 0.43 -> 0.98 | Helped |
| add dynamic verification (ablation) | Swift · CNN/DM | speedup: 1.342× -> 1.560× | Helped |
| add dynamic verification (ablation) | Swift · CNN/DM | mean generated length: 4.39 -> 5.82 | Helped |
| add the layer-set optimization phase (ablation) | Swift · CNN/DM | speedup: 1.224× -> 1.560× | Helped |
| set context window γ to 16 instead of 32 (ablation) | Swift · CNN/DM | speedup: 1.560× -> 1.341× | Hurt |
| set context window γ to 64 instead of 32 (ablation) | Swift · CNN/DM | speedup: 1.560× -> 1.552× | No clear effect |
| set context window γ to 128 instead of 32 (ablation) | Swift · CNN/DM | speedup: 1.560× -> 1.550× | No clear effect |
Surprising findings
- Uniformly skipping about half of the layers during drafting already yields about 1.2x speedup, contrary to the belief that skipped-layer selection must be carefully curated.
- Larger models show greater layer sparsity, with optimal skip ratio and speedup increasing with model size.
Limitations
- Speedup depends on layer-skip ratio and task; the current implementation has room for further optimization (expected speedups exceed measured ones).
- Evaluated mainly on LLaMA-2/CodeLLaMA and a few other backbones; scaling to very large models (e.g., 175B) left to future work.
- Batch size fixed at 1 in experiments.
Experimental setup
Datasets: CNN/Daily Mail GSM8K TinyStories HumanEval Alpaca WMT14 DE-EN Nature Questions
Benchmarks: CNN/Daily Mail GSM8K TinyStories HumanEval
Metrics: wall-time speedup mean generated length token acceptance rate pass@1 pass@10 tokens/s
Baselines: Parallel Decoding Lookahead Decoding Self-SD LayerSkip Vanilla autoregressive decoding
Builds on: Self-SD speculative sampling Lookahead decoding Parallel decoding
Closest papers by meaning
- Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding (NeurIPS 2025 · similarity 0.92)
- SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.92)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.92)
- PEARL: Parallel Speculative Decoding with Adaptive Draft Length (ICLR 2025 · similarity 0.92)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.92)
- TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.