Research map / Large language models / Reasoning and chain of thought
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CTC-drafter, a speculative decoding draft module trained with sequence-level CTC loss on distilled ShareGPT data, achieving higher acceptance length and speedup over Medusa and Hydra on Vicuna models.
Problem
Speculative decoding draft models that predict multiple future tokens non-autoregressively ignore dependencies between draft tokens, lowering acceptance rate and thus LLM inference speedup.
Approach
Train the draft module with CTC loss so that probability mass over blank/repeat-expanded alignments captures inter-token dependencies; at inference, CTC transform removes blanks and duplicates from candidate sequences and masks the corresponding attention positions before verification.
Key results
- On MT-bench with Vicuna-7B, CTC-drafter reaches 2.78x speedup and 3.56 accepted tokens per step vs. Medusa 2.13x / 2.58 and Hydra 2.36x / 3.04.
- On GSM8K with Vicuna-13B, CTC-drafter achieves 2.66x speedup vs. Medusa 2.21x.
- Ablation on Vicuna-7B: replacing Medusa's linear head with a transformer layer and CTC loss without CTC transform gives 2.25x / 3.02 accepted tokens, versus 2.13x / 2.58 for Medusa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CTC-drafter instead of Medusa | Vicuna-7B speculative decoding · MT-bench | speedup: 2.13× -> 2.78× | Helped |
| use CTC-drafter instead of Hydra | Vicuna-7B speculative decoding · MT-bench | speedup: 2.36× -> 2.78× | Helped |
| use CTC-drafter instead of Medusa | Vicuna-13B speculative decoding · MT-bench | speedup: 1.97× -> 2.52× | Helped |
| use CTC-drafter instead of Medusa | Vicuna-33B speculative decoding · MT-bench | speedup: 1.93× -> 2.20× | Helped |
| use CTC-drafter instead of Medusa | Vicuna-7B speculative decoding · GSM8K | speedup: 2.33× -> 2.43× | Helped |
| use CTC-drafter instead of Medusa | Vicuna-13B speculative decoding · GSM8K | speedup: 2.21× -> 2.66× | Helped |
| use CTC-drafter instead of Medusa | Vicuna-33B speculative decoding · GSM8K | speedup: 2.10× -> 2.16× | Helped |
| replace linear draft head and cross-entropy loss with transformer draft layer and CTC loss (ablation) | Medusa verification speculative decoding · MT-bench | accepted tokens per decoding step: 2.58 -> 3.02 | Helped |
| replace Medusa verification with CTC transform and attention-map modification (ablation) | CTC-drafter · MT-bench | speedup: 2.25× -> 2.78× | Helped |
| replace Medusa verification with CTC transform and attention-map modification (ablation) | CTC-drafter · MT-bench | accepted tokens per decoding step: 3.02 -> 3.56 | Helped |
| replace Medusa's linear draft head with a transformer draft layer (ablation) | speculative decoding draft module · inference time consumption | draft model time share: 3.71% -> 14.93% | Hurt |
| add CTC transform for candidate sequences (ablation) | CTC-drafter · inference time consumption | CTC transform time share: 5.36% | Hurt |
Surprising findings
- Speedup and accepted tokens decline as base model size grows from 7B to 33B for CTC-drafter (2.20x at 33B on MT-bench), attributed to the capacity gap between draft module and base model.
- Adding transformer layer and CTC loss without the CTC transform (blank/repeat removal) lowers performance relative to full CTC-drafter, showing the transform is needed.
Limitations
- Draft module adds extra time overhead (draft share rises from 3.71% to 14.93%) and CTC transform adds 5.36% decoding time.
- Performance degrades on larger base models (33B) and on GSM8K for 7B, and evaluation is limited to Vicuna and LLaMA-2-Chat families.
- Only Medusa-1 style training (frozen base model) is implemented for comparison; no evaluation on other LLM families.
Experimental setup
Datasets: ShareGPT MT-bench GSM8K
Benchmarks: MT-bench GSM8K
Metrics: speedup ratio average accepted tokens per decoding step
Baselines: Vanilla autoregressive decoding Medusa Hydra
Builds on: Medusa Hydra Connectionist Temporal Classification Speculative Decoding Blockwise Decoding
Closest papers by meaning
- Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning (ICLR 2026 · similarity 0.93)
- Cascade Speculative Drafting for Even Faster LLM Inference (NeurIPS 2024 · similarity 0.93)
- Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs (ICLR 2026 · similarity 0.92)
- Towards Optimal Multi-draft Speculative Decoding (ICLR 2025 · similarity 0.92)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.92)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.