Research map / Large language models / Reasoning and chain of thought
Talon: Breaking the Synchronization Barrier in Speculative Decoding with Hybrid Model-based and Retrieve-based Drafting
Xiangxiang Gao, Weisheng Xie, , Lixin, Xuwei Fang, Chen Hang, Changqun Li, Yuhan Lin, Xiaolong Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Talon, a speculative decoding framework that hybridizes model-based and retrieval-based drafting to remove the synchronization barrier between draft generation and verification.
Problem
Speculative decoding's draft-then-verify pipeline suffers from a mutual waiting problem, where draft generation and verification depend sequentially on each other, limiting LLM inference acceleration.
Approach
Combines model-based drafting with retrieval-based drafting in a hybrid scheme so that drafting and verification no longer block each other, breaking the synchronization barrier.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace EAGLE-3 with Talon (async hybrid drafting) | Vicuna-13B · MT-Bench (greedy) | speedup: 5.58x -> 5.84x | Improved |
| replace EAGLE-3 with Talon (async hybrid drafting) | Vicuna-13B · CNN/DM (greedy) | speedup: 5.01x -> 4.67x | Hurt |
| replace EAGLE-3 with Talon (async hybrid drafting) | LLaMA-3.1 8B · MT-Bench (greedy) | speedup: 4.40x -> 4.99x | Improved |
| replace EAGLE-3 with Talon (async hybrid drafting) | LLaMA-3.1 8B · Alpaca (greedy) | speedup: 4.82x -> 4.98x | No clear effect |
| replace EAGLE-3 with Talon (async hybrid drafting) | DeepSeek-R1-Distill-LLaMA 8B · MT-Bench (greedy) | speedup: 4.05x -> 4.93x | Improved |
| replace EAGLE-3 with Talon (async hybrid drafting) | Vicuna-13B · MT-Bench (greedy) | average acceptance length: 6.65 -> 8.11 | Improved |
| replace EAGLE-3 with Talon (async hybrid drafting) | DeepSeek-R1-Distill-LLaMA 8B · GSM8K (greedy) | speedup: 5.01x -> 5.43x | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- EAGLE-3 — EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025) · better on 17 of 18 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Experimental setup
Builds on: speculative decoding
Closest papers by meaning
- TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.93)
- PEARL: Parallel Speculative Decoding with Adaptive Draft Length (ICLR 2025 · similarity 0.93)
- Falcon: Faster and Parallel Inference of Large Language Models Through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree (AAAI 2025 · similarity 0.92)
- SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.92)
- Block Verification Accelerates Speculative Decoding (ICLR 2025 · similarity 0.91)
- Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
