Research map / Large language models / Reasoning and chain of thought
Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Kangaroo proposes lossless self-speculative decoding that drafts with a shallow sub-network of the target LLM plus a lightweight trained adapter and a token-level confidence-based early exit, achieving up to 2.04x walltime speedup on Spec-Bench with Vicuna.
Problem
Speculative decoding normally needs a separately trained draft model, which is costly; existing self-drafting methods (Medusa, REST, Lookahead) trade off token acceptance rate against drafting latency.
Approach
Use the first l layers of the target LLM plus its frozen LM head as the draft model, bridged by a small adapter (one multi-head attention layer and two norms, no FFN), and add a second early exit that halts drafting when the draft's top-1 probability falls below a threshold, extended to dynamic token trees.
Key results
- Kangaroo achieves average walltime speedups of 1.72x on Vicuna-7B and 1.65x on Vicuna-13B on Spec-Bench with dynamic tree decoding, with peak subtask speedup 2.04x on Vicuna-7B MT-bench/QA-style tasks.
- Kangaroo adapter has 67M parameters versus 591M for Medusa heads (about 11.3%), yet Kangaroo w/o tree reaches 1.50x vs Medusa w/o tree 1.41x on Vicuna-7B.
- Training the adapter for Vicuna-7B takes about 24 hours on 8 NVIDIA V100 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dynamic tree verification instead of single-sequence decoding | Kangaroo on Vicuna-7B · Spec-Bench average | walltime speedup: 1.50× -> 1.72× | Helped |
| use dynamic tree verification instead of single-sequence decoding | Kangaroo on Vicuna-13B · Spec-Bench average | walltime speedup: 1.44× -> 1.65× | Helped |
| add the target model's LM Head to the Kangaroo adapter (ablation) | Kangaroo on Vicuna-7B · Spec-Bench average | walltime speedup: 1.50 × -> 1.44 × | Hurt |
| use a 1-layer Transformer adapter (ablation) | Kangaroo on Vicuna-7B · Spec-Bench average | walltime speedup: 1.50 × -> 1.37 × | Hurt |
| replace the attention adapter with an MLP-only adapter (ablation) | Kangaroo on Vicuna-7B · Spec-Bench average | walltime speedup: 1.50 × -> 1.22 × | Hurt |
| use adjusted rather than original confidence for early stopping at temperature 0.2 (ablation) | Kangaroo on Vicuna-7B · Spec-Bench average | walltime speedup: 1.48× -> 1.34× | Hurt |
Surprising findings
- Fixed-step drafting (threshold 0) maximizes compression rate but yields suboptimal end-to-end walltime speedup, showing acceptance rate alone is misleading.
- Adding the LM head to the adapter (198M params) reduced speedup relative to the lighter FFN-free adapter (1.44x vs 1.50x).
Limitations
- Effectiveness depends on the target model and task; representation gap may persist on complex tasks.
- No error bars or statistical significance reported.
Experimental setup
Datasets: ShareGPT Spec-Bench
Benchmarks: Spec-Bench
Metrics: walltime speedup ratio compression rate consistent token acceptance rate
Baselines: Lookahead Medusa REST SpS Draft & Verify
Builds on: speculative decoding Medusa early exiting Draft & Verify
Closest papers by meaning
- SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.91)
- polybasic Speculative Decoding Through a Theoretical Perspective (ICML 2025 · similarity 0.90)
- LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding (ICML 2026 · similarity 0.90)
- SEED: Self-Speculative Decoding via Implicit Encoder–Decoder (NeurIPS 2026 · similarity 0.90)
- Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding (ICML 2025 · similarity 0.90)
- GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.