Research map / Large language models / Reasoning and chain of thought
Steering Pretrained Drafters During Speculative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SD² (steered speculative decoding), which computes a steering vector from verifier hidden states and injects it into a pretrained drafter's MLPs to raise token acceptance without changing the verifier.
Problem
Pretrained independent drafters align poorly with verifiers offline (distillation generalizes poorly out of distribution), so block efficiency is low when verification cost dominates latency.
Approach
A steering vector computed from the verifier's high/middle/low-level hidden states at the last rejected position is injected as a bias into every MLP up-projection of a pretrained drafter, which is fine-tuned jointly with a frozen verifier.
Key results
- With Vicuna 1.3 13B / Llama 160M at T=1, SD² reaches mean block efficiency 2.96 vs 1.88 pretrained and 2.45 distilled (mean alpha 1.61).
- Up to 35% more accepted tokens and up to 22% higher throughput than independent drafters across tasks and models (per abstract).
- Average speedup of +16.3% at T=1 and +19.5% at T=0 over pretrained drafter; roughly 2x distillation's additive speedup under sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add steering vector from verifier hidden states to pretrained drafter (SD²) vs. pretrained drafter, block efficiency | Vicuna 1.3 13B / Llama 160M · Mean over UltraChat, HumanEval, XSum, Alpaca, GSM8K, T=1, k=8 | block efficiency τ: +1.08 (1.88 to 2.96) | Helped |
| Add steering vector to pretrained drafter (SD²) vs. distillation | Qwen3 14B / Qwen3 0.6B · Mean over 5 tasks, T=1, k=8 | block efficiency τ: 4.26 vs 3.97 | Helped |
| Add steering vector to pretrained drafter (SD²) vs. distillation | Llama 3.1 8B / Llama 3.2 1B · Mean over 5 tasks, T=1, k=8 | block efficiency τ: 5.00 vs 4.78 | Helped |
| Add steering vector to pretrained drafter (SD²) vs. pretrained drafter | Qwen3 8B / Qwen3 0.6B · HumanEval, T=1, k=8 | block efficiency τ: 5.18 vs 5.18 | No clear effect |
| Add steering vector to pretrained drafter (SD²) vs. pretrained drafter | Llama 3.1 8B / Llama 3.2 1B · GSM8K, T=1, k=8 | block efficiency τ: 5.44 vs 5.62 | Hurt |
| Add steering vector to pretrained drafter (SD²) vs. pretrained drafter | Vicuna 1.3 13B / Llama 160M · Mean over 5 tasks, T=0 greedy, k=8 | block efficiency τ: 3.27 vs 2.36 | Helped |
| Add steering vector to pretrained drafter (SD²) vs. pretrained drafter | Llama 3.1 8B / Llama 3.2 1B · Mean over 5 tasks, T=0 greedy, k=8 | block efficiency τ: 5.31 vs 5.12 | Helped |
| Distillation of drafter to verifier vs. pretrained drafter | Qwen3 14B / Qwen3 0.6B and Llama 3.1 8B / Llama 3.2 1B · HumanEval and GSM8K, T=1 | block efficiency τ: e.g. GSM8K 5.22 vs 5.62 for Llama 3.2 1B | Hurt |
| Freeze drafter and train only steering mechanism (vs. full fine-tuning with steering) (ablation) | Vicuna 1.3 7B / Llama 160M · Ablation, block efficiency | block efficiency τ | Hurt |
| Steering via bias in MLP up-projection (adopted) vs. simpler post-MLP bias or 2-layer MLP bias (ablation) | Vicuna 1.3 7B / Llama 160M · Ablation, block efficiency | block efficiency τ | Helped |
| Random offset training vs. blocked offset training of steering (ablation) | Vicuna 1.3 7B / Llama 160M · Ablation, training offset | block efficiency τ | No clear effect |
| Add steering vector to pretrained drafter vs. distillation, speedup | Vicuna 1.3 13B / Llama 160M · Mean over tasks, T=1 | speedup over pretrained (α): +61% speedup | Helped |
Surprising findings
- Steering the frozen drafter alone (without fine-tuning) already roughly doubles accepted tokens over the unaligned baseline.
- Distillation degrades performance on tasks unseen in training (GSM8K, HumanEval for Qwen/Llama), while steering does not.
Limitations
- Performance depends heavily on training data composition; strongest on in-distribution data.
- Requires access to verifier hidden states, which is hard with remote verifiers.
- Little to no speedup over already well-aligned drafters such as Llama 3.1 pairs; only chain drafting evaluated.
Experimental setup
Datasets: UltraChat_200k ShareGPT
Benchmarks: UltraChat_200k HumanEval XSum Alpaca GSM8K
Metrics: block efficiency speedup (throughput)
Baselines: Pretrained independent drafter Distilled drafter EAGLE-3*
Builds on: speculative sampling EAGLE-3 distillation for speculative decoding (DistillSpec) SwiGLU
Closest papers by meaning
- Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning (ICLR 2026 · similarity 0.92)
- GRIFFIN: Effective Token Alignment for Faster Speculative Decoding (NeurIPS 2025 · similarity 0.92)
- SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting (NeurIPS 2026 · similarity 0.91)
- D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting (NeurIPS 2026 · similarity 0.91)
- Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.91)
- Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.