Research map / Large language models / Reasoning and chain of thought
SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SuffixDecoding, a model-free speculative decoding method that uses global and per-request suffix trees with adaptive speculation length to accelerate repetitive agentic LLM workloads, reaching up to 5.3x speedup.
Problem
Speculative decoding for LLM inference is inefficient on agentic workloads, which produce long, repetitive, highly predictable token sequences; existing draft-model methods have overhead and model-free methods (e.g., PLD) lack adaptive speculation length.
Approach
Cache prompt and previous output tokens in suffix trees, pattern-match the current suffix, greedily grow a speculation tree scored by frequency-based acceptance estimates, and set the speculation budget adaptively as alpha times the matched pattern length; optionally fall back to EAGLE-3 when the score is low.
Key results
- On AgenticSQL, SuffixDecoding achieves a 5.3x mean speedup over vanilla decoding, 2.8x over EAGLE-2/3 and 1.9x over Token Recycling.
- On SWE-Bench, SuffixDecoding obtains a 2.5x mean speedup and 7.8 mean accepted tokens per step, versus 1.7x speedup for PLD.
- On Spec-Bench, the hybrid SuffixDecoding + EAGLE-3 reaches 2.5x speedup, above EAGLE-3 (2.4x) and Token Recycling (2.2x).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SuffixDecoding (linear) instead of vanilla decoding | Llama-3.1-8B-Instruct · AgenticSQL | speedup over vanilla decoding: 5.345 | Helped |
| Use SuffixDecoding (tree) instead of vanilla decoding | Llama-3.1-8B-Instruct · SWE-Bench | speedup over vanilla decoding: 2.433 | Helped |
| Use hybrid SuffixDecoding with EAGLE-3 fallback | Llama-3.1-8B-Instruct · Spec-Bench | speedup over vanilla decoding: 2.500 | Helped |
| Use SuffixDecoding alone (τ = 0) instead of hybrid fallback (ablation) | SuffixDecoding · AgenticSQL | speedup: 5.35 vs 3.95 | Helped |
| Use SuffixDecoding alone (τ = 0) rather than EAGLE-3 fallback (ablation) | SuffixDecoding · Spec-Bench | speedup: 1.66 vs 2.50 | Hurt |
| Add the global suffix tree to the per-request suffix tree (ablation) | SuffixDecoding · AgenticSQL tasks other than Enrich and Extract | speedup | Helped |
| Add the per-request suffix tree to the global suffix tree (ablation) | SuffixDecoding · AgenticSQL Enrich and Extract | speedup | Hurt |
| Use the global suffix tree rather than the per-request tree (ablation) | SuffixDecoding · AgenticSQL tasks except Combine | speedup | Helped |
| Increase the global suffix tree with more prior output examples (ablation) | SuffixDecoding · WildChat and Magicoder | speedup | Helped |
| Increase the global suffix tree with more prior output examples (ablation) | SuffixDecoding · WildChat and Magicoder | acceptance rate | No clear effect |
| Adapt the global suffix tree online with SpiderSQL outputs after distribution shift (ablation) | SuffixDecoding trained on WildChat · SpiderSQL | speedup: 1.5× speedup ... 2.6× speedup | Helped |
Surprising findings
- Suffix-only SuffixDecoding underperforms EAGLE-2/3 and Token Recycling on open-ended Spec-Bench, so it is not universally superior.
- Acceptance rate on WildChat and Magicoder stays roughly constant as the suffix tree grows even though speedup keeps increasing.
Limitations
- Gains depend on repetitive workloads; open-ended chat sees little benefit without fallback.
- AgenticSQL is proprietary, so the main agentic results are hard to reproduce.
- Global suffix tree memory grows linearly and needs eviction for very long deployments.
Experimental setup
Datasets: SWE-Bench Verified AgenticSQL traces Spec-Bench WildChat Magicoder-Evol-Instruct-110K Spider / DAIL-SQL SpiderSQL prompts
Benchmarks: SWE-Bench AgenticSQL Spec-Bench WildChat Magicoder
Metrics: speedup over vanilla decoding mean accepted tokens per step acceptance rate time per output token end-to-end task completion time
Baselines: EAGLE-2 EAGLE-3 Prompt Lookup Decoding Token Recycling Medusa
Builds on: speculative decoding Prompt Lookup Decoding Token Recycling EAGLE-3 vLLM
Closest papers by meaning
- Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.90)
- Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.90)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.89)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.89)
- Online Speculative Decoding (ICML 2024 · similarity 0.89)
- MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.