Research map / Large language models / Reasoning and chain of thought
RepSpec: Structural Re-parameterized Draft Model Training for Speculative Decoding
Feiye Huo, Jianchao Tan, Jiahao Liu, Zixu Jiang, Jiacheng Li, Jingang Wang, Xunliang Cai, Shengli Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RepSpec, a structural re-parameterized training scheme for speculative decoding draft models (only the abstract was available).
Problem
Autoregressive LLM inference is memory-bound and slow; speculative decoding speedup is fundamentally limited by the quality of the small draft model.
Approach
Uses structural re-parameterization to train the draft model, so that extra training-time structure can be folded back into a plain draft architecture at inference. Details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train draft with pure linear re-parameterization (Pre+Bypass) | EAGLE-3 on LLaMA-3.1-8B, chain · avg of 6 tasks | acceptance length: 3.35 -> 3.60 | Improved |
| train draft with pure linear re-parameterization | EAGLE-1 on LLaMA-3.1-8B, chain · avg of 6 tasks | decoding speed: 48 -> 52 | Improved |
| use hybrid (nonlinear bypass) instead of linear | EAGLE-3 on LLaMA-3.1-8B, chain · avg of 6 tasks | decoding speed: 68 -> 65 | Hurt |
| use hybrid instead of linear on larger target | EAGLE-1 on LLaMA-2-13B, tree · avg of 6 tasks | decoding speed: 78 -> 80 | Improved |
| apply RepSpec Linear to Medusa | Medusa on Vicuna-1.3-7B · avg of 6 tasks | decoding speed: 44 -> 46 | Improved |
| apply RepSpec Linear to Hydra acceptance length | Hydra on Vicuna-1.3-7B · avg of 6 tasks | acceptance length: 3.59 -> 3.60 | No clear effect |
| re-parameterize the embedding layer (ablation) | EAGLE-1 draft · 6 benchmarks | acceptance length: MT 2.51 -> 2.55; GSM8K 2.74 -> 2.70 | Mixed |
| re-parameterize attention and MLP linear layers (ablation) | EAGLE-1 draft · MT-bench validation | acceptance rate | Improved |
| use more than one Pre and one Bypass block (ablation) | pure linear RepSpec · MT-bench validation | acceptance rate | No clear effect |
| double decoder layers vs hybrid re-parameterization (ablation) | EAGLE-1 draft · MT-bench | acceptance length: 2.77 -> 2.79 (0.47B vs 0.40B params) | Improved |
| choose ReLU instead of GeLU for bypass nonlinearity (ablation) | hybrid RepSpec · MT-bench | decoding speed: tau 2.85 vs 2.85; speed 51 vs 50 | No clear effect |
| halve mid_feature in hybrid (ablation) | hybrid RepSpec · MT-bench | acceptance length: 2.85 -> 2.78; speed 51 -> 53 | Mixed |
Experimental setup
Builds on: speculative decoding
Closest papers by meaning
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.91)
- Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration (NeurIPS 2024 · similarity 0.90)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.90)
- LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding (ICML 2026 · similarity 0.90)
- Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.90)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
