Research map / Large language models / Reasoning and chain of thought
Falcon: Faster and Parallel Inference of Large Language Models Through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Falcon, a semi-autoregressive speculative drafter trained with Coupled Sequential Glancing Distillation and a custom relaxed-causal decoding tree, achieving 2.91x-3.51x lossless speedup on Vicuna and LLaMA2-Chat.
Problem
Speculative decoding for LLMs faces a trade-off between low drafting latency (parallel drafting) and high speculation accuracy, since semi-autoregressive drafters ignore inter-token dependencies within a block.
Approach
Coupled Sequential Glancing Distillation trains a SAR drafter by replacing contiguous token and feature spans preceding the prediction targets with LLM ground truth, plus a KL/CE distillation loss; a custom relaxed-causal decoding tree lets the drafter run multiple forward passes and produce k tokens per pass.
Key results
- Falcon achieves 2.91x-3.51x speedup across MT-Bench, HumanEval and GSM8K on Vicuna and LLaMA2-Chat, e.g. 3.10x on Vicuna-7B MT-Bench versus 2.82x for Eagle.
- On Vicuna-7B MT-Bench, tree attention raises speedup from 2.06x to 2.65x and CSGD from 2.06x to 2.27x; combined gives 3.10x.
- Falcon's acceptance rate on Vicuna-7B MT-Bench is 77.64% versus 74.59% for Eagle and 60.53% for Medusa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Falcon instead of Eagle | Vicuna-7B speculative decoding · MT-Bench | speedup: 2.82x -> 3.10x | Helped |
| use Falcon instead of Eagle | Vicuna-13B speculative decoding · HumanEval | speedup: 2.97x -> 3.12x | Helped |
| use Falcon instead of Eagle | LLaMA2-Chat-7B speculative decoding · GSM8K | speedup: 2.89x -> 2.91x | Helped |
| use Falcon instead of Eagle | LLaMA2-Chat-13B speculative decoding · HumanEval | speedup: 3.25x -> 3.51x | Helped |
| add tree attention rather than using an equal-height chain (ablation) | Falcon · MT-Bench, Vicuna-7B | speedup: 2.65x -> 3.10x | Helped |
| add tree attention rather than using an equal-height chain (ablation) | Falcon · MT-Bench, Vicuna-7B | acceptance rate (α): 74.37 -> 77.64 | Helped |
| add tree attention rather than using an equal-height chain (ablation) | Falcon · MT-Bench, Vicuna-7B | average acceptance length (τ): 2.78 -> 3.34 | Helped |
| add CSGD training (ablation) | Falcon · MT-Bench, Vicuna-7B | speedup: 2.65x -> 3.10x | Helped |
| add CSGD training (ablation) | Falcon · MT-Bench, Vicuna-7B | average acceptance length (τ): 2.78 -> 3.34 | Helped |
| add CSGD training (ablation) | Falcon · MT-Bench, Vicuna-7B | acceptance rate (α): 74.37 -> 77.64 | Helped |
| increase k from 2 to 3 (ablation) | Falcon · MT-Bench, Vicuna-7B | speedup: 3.10x -> 2.96x | Hurt |
| increase k from 2 to 3 (ablation) | Falcon · MT-Bench, Vicuna-7B | average acceptance length (τ): 3.34 -> 2.52 | Hurt |
Surprising findings
- Increasing the number of tokens per forward pass k from 2 to 4 reduces speedup only modestly (3.10x to 2.86x) while still exceeding Eagle.
Limitations
- Tree structures are hand-designed and reported as model- and context-dependent.
- Speedup gains over Eagle are modest on some settings (e.g. 1.01x-1.24x) and Vicuna-7B tau is lower than Eagle's.
Experimental setup
Datasets: ShareGPT
Benchmarks: MT-Bench HumanEval GSM8K
Metrics: wall-time speedup ratio acceptance rate average acceptance length
Baselines: SpS PLD Lookahead Medusa Eagle
Builds on: EAGLE Medusa SpecInfer Glancing Transformer Speculative sampling
Closest papers by meaning
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.91)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.91)
- DFlash: Block Diffusion for Flash Speculative Decoding (ICML 2026 · similarity 0.91)
- TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding (NeurIPS 2026 · similarity 0.91)
- PEARL: Parallel Speculative Decoding with Adaptive Draft Length (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.