Research map / Large language models / Reasoning and chain of thought
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GliDe, a draft model that cross-attends to the frozen target LLM's KV cache, and CaPE, a confidence-aware expansion of draft proposals for verification, to accelerate speculative decoding.
Problem
Speculative decoding speedup depends on the acceptance rate of draft-model proposals, and existing draft models are misaligned with the target model and single-sequence proposals limit acceptance.
Approach
The draft model is a lightweight transformer with an added cross-attention sub-layer that reuses the frozen target model's KV cache (trained with a block-wise attention mask to simulate KV delay). CaPE expands each proposal position with a number of extra top-ranked tokens chosen by draft confidence, verified in one pass with a special causal mask.
Key results
- GliDe accelerates Vicuna models up to 2.17x on walltime, and GliDe+CaPE reaches up to 2.61x.
- GliDe achieves acceptance rates of roughly 55-70% versus about 30-60% for LLaMA baseline draft models, with expected speedups 1.67-2.24.
- On MT-Bench with Vicuna-7B, CaPE raises decoding speed from 68.4 to 70.6 tokens/sec relative to fixed-size expansion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add cross-attention to reuse the target model's KV cache (ablation) | Vanilla Drafter · GSM8K, Finance-Alpaca, Spider, and Code-Search-Python | acceptance rate: 5.4 and 23.5 percentage points | Helped |
| Add cross-attention to reuse target KV cache (ablation) | GliDe draft model · GSM8K, Finance-Alpaca, Spider, and Code-Search-Python | acceptance rate | Helped |
| Use confidence-aware proposal expansion instead of fixed-size expansion sets of 4 (ablation) | GliDe · MT-Bench | decoding speed (tokens/sec): 70.6 tokens/sec to 68.4 tokens/sec | Helped |
| Add sequence-level distillation (ablation) | GliDe · GSM8K | acceptance rate (%): 64.8 -> 65.0 | Helped |
| Add sequence-level distillation (ablation) | GliDe · Finance-Alpaca | acceptance rate (%): 63.2 -> 63.9 | Helped |
| Add sequence-level distillation (ablation) | GliDe · Spider | acceptance rate (%): 55.7 -> 56.0 | Helped |
| Add sequence-level distillation (ablation) | GliDe · Code-Search-Python | acceptance rate (%): 67.0 -> 67.5 | Helped |
| Use beam search with beam size 4 instead of plain GliDe speculation (ablation) | GliDe · MT-Bench | walltime speedup | Hurt |
| Use beam search instead of confidence-aware proposal expansion (ablation) | GliDe · MT-Bench | speculation walltime: 5.5ms to 10.9ms; a difference of 5.4ms | Hurt |
| Add CaPE to GliDe | GliDe · MT-Bench | walltime speedup: 2.50x and 2.61x | Helped |
Surprising findings
- Beam-search-based proposal expansion (GliDe+BeamSearch) is slower than GliDe alone due to speculation overhead from copying and sorting.
- A 12-layer LLaMA-160m draft model can reduce decoding speed, so wider and shallower draft models are preferred.
Limitations
- Draft model must be retrained for each new target LLM and tokenizer, so it is not plug-and-play.
- Training cost is dominated by forwarding the target model to obtain KV caches.
- Batch serving shows OOM at larger batch sizes and long prompts; no batch-specific algorithm is designed.
Experimental setup
Datasets: SlimPajama-6B ShareGPT GSM8K Finance-Alpaca Spider Code-Search-Python MT-Bench
Benchmarks: GSM8K Finance-Alpaca Spider Code-Search-Python MT-Bench
Metrics: acceptance rate expected speedup cost coefficient walltime speedup tokens per second
Baselines: LLaMA-68m LLaMA-160m LLaMA-45m Medusa GliDe+BeamSearch Vanilla Drafter
Builds on: speculative sampling Medusa SpecInfer tree verification LLaMA draft models
Closest papers by meaning
- SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.91)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.90)
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.90)
- PEARL: Parallel Speculative Decoding with Adaptive Draft Length (ICLR 2025 · similarity 0.90)
- SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding (ICML 2026 · similarity 0.90)
- Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.