Research map / Large language models / Reasoning and chain of thought
GRIFFIN: Effective Token Alignment for Faster Speculative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRIFFIN, a speculative decoding draft-model framework that masks misaligned tokens during multi-step training (top-k alignment) and adds a Token-Guided Fusion module, improving acceptance length and speedup over EAGLE-2 and HASS.
Problem
Speculative decoding draft models (EAGLE, EAGLE-2, HASS) suffer from token misalignment between training (ground-truth tokens) and decoding (draft tokens), which compounds over multiple forward passes and limits acceptance length and speedup.
Approach
Use a dynamic loss mask that excludes tokens whose ground truth is not in the draft model's top-k predictions (with cumulative masks across forward passes), plus a Token-Guided Fusion module that re-injects token embeddings into the draft model's feature fusion, so training and decoding are aligned.
Key results
- On LLaMA3 8B at temperature 0, GRIFFIN averages 3.35x speedup and 5.38 acceptance length versus 3.12x and 5.13 for HASS (about 7% speedup gain).
- On Vicuna 7B HumanEval at temperature 1, GRIFFIN reaches 4.12x speedup versus 3.62x for HASS (about 13.8% gain).
- Ablation on LLaMA2-Chat 7B: removing both TAT and TAD drops average acceptance length from 5.44 to 4.82 at temperature 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token-alignable training (TAT) (ablation) | GRIFFIN · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 3.10 -> 3.28 | Helped |
| Add token-alignable training (TAT) (ablation) | GRIFFIN · MT-Bench, HumanEval, GSM8K average; T=0 | acceptance length: 5.18 -> 5.44 | Helped |
| Add token-alignable draft model (TAD) (ablation) | GRIFFIN · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 3.16 -> 3.28 | Helped |
| Add token-alignable draft model (TAD) (ablation) | GRIFFIN · MT-Bench, HumanEval, GSM8K average; T=0 | acceptance length: 5.25 -> 5.44 | Helped |
| Add both TAT and TAD (ablation) | GRIFFIN · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 2.89 -> 3.28 | Helped |
| Use alignment top-k=3 rather than top-k=1 (ablation) | GRIFFIN · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 3.21 -> 3.28 | Helped |
| Use alignment top-k=3 rather than top-k=10 (ablation) | GRIFFIN · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 3.21 -> 3.28 | Helped |
| Increase TAT training steps from 1 to 3 (ablation) | GRIFFIN · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 3.10 -> 3.28 | Helped |
| Increase TAT training steps from 3 to 5 (ablation) | GRIFFIN · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 3.28 -> 3.30 | Helped |
| Replace TGF secondary token embedding input with raw features (ablation) | GRIFFIN TGF · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 3.28 -> 2.80 | Hurt |
| Replace TGF secondary token embedding input with initial fused features (ablation) | GRIFFIN TGF · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 3.28 -> 3.10 | Hurt |
| Use GRIFFIN rather than HASS | speculative decoding · MT-Bench, HumanEval, GSM8K average; T=0 | speedup ratio: 3.17 -> 3.28 | Helped |
Surprising findings
- Naively substituting draft tokens for ground-truth tokens during training significantly reduces acceptance length, so token alignment must be handled via masking instead.
- Replacing token embeddings in TGF with raw features reduces acceptance length by 0.83, showing gains are not just from extra parameters.
Limitations
- Multi-step training adds training overhead compared to EAGLE.
- Speedup gains are smaller for Mixtral-8x7B due to MoE verification overhead.
- Qwen2 speedup is lower due to its larger vocabulary LM head.
Experimental setup
Datasets: ShareGPT
Benchmarks: MT-Bench HumanEval GSM8K
Metrics: speedup ratio acceptance length
Baselines: SPS PLD Lookahead Medusa EAGLE EAGLE-2 EAGLE-3 FSPAD HASS
Builds on: EAGLE EAGLE-2 HASS Medusa
Closest papers by meaning
- Steering Pretrained Drafters During Speculative Decoding (AAAI 2026 · similarity 0.92)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.91)
- Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.91)
- Online Speculative Decoding (ICML 2024 · similarity 0.91)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.