Research map / Architectures and efficiency / RNNs and normalization
EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EasySpec, a training-free plug-in that parallelizes draft-model attention layers across GPUs (fuzzy speculation) and periodically recalibrates the draft KV cache with a single token-parallel pass (bonus calibration), speeding up speculative decoding on multi-GPU systems.
Problem
In multi-GPU tensor-parallel speculative decoding, the small draft model's optimal TP size is small, so GPUs sit idle during drafting because its layers must run sequentially.
Approach
Break inter-layer data dependencies in the draft model by feeding attention layers earlier hidden states (exploiting high cosine similarity between hidden states) so they can run concurrently on different GPUs; then after each draft-verify iteration, recompute precise draft KV cache in one token-parallel forward pass to stop error accumulation. Lossless because the verification step still uses the
Key results
- EasySpec reaches up to 4.17x total speedup over vanilla decoding (Qwen2-72B with 7B drafter, HumanEval, temperature 0.8), versus 3.18x for +tree baseline.
- Drafting stage accelerated by up to 1.62x with at most 7% drop in speculation acceptance accuracy relative to +tree.
- On Spec-Bench with Llama-3-70B-Instruct, EasySpec (1B drafter) achieves higher mean throughput than EAGLE-2 across tasks, with acceptance ~0.8 vs <0.4 for EAGLE-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EasySpec instead of +tree | Llama-3-70B-Instruct and Qwen2-72B-Instruct speculative decoding · MMLU, HumanEval, MATH, IFEval, and MGSM at temperature=0.8 | total acceleration ratio: 3.61x vs 2.89x; 3.94x vs 3.30x (MMLU); 3.90x vs 2.98x; 4.17x vs 3.44x (HumanEval); 3.86x vs 2.97x; 4.13x vs 3.39x (MATH) | Helped |
| Use EasySpec instead of +tree | Qwen2.5-Coder-32B-Instruct (7B drafter) · HumanEval | total running time per 100 tokens: 3.38 -> 2.66 | Helped |
| Use EasySpec instead of +tree | Qwen2-Math-72B-Instruct (7B drafter) · MATH | total running time per 100 tokens: 3.55 -> 2.91 | Helped |
| Parallelize full layers rather than only attention layers (ablation) | EasySpec · MMLU, Llama-3-70B-Instruct (1B drafter), LP size 2 | token throughput: 34.37 -> 29.71 | Hurt |
| Parallelize full layers rather than only attention layers (ablation) | EasySpec · MMLU, Llama-3-70B-Instruct (1B drafter), LP size 2 | acceptance rate: 0.77 -> 0.58 | Hurt |
| Parallelize full layers rather than only attention layers (ablation) | EasySpec · MMLU, Qwen2-72B-Instruct (1.5B drafter), LP size 2 | token throughput: 29.94 -> 27.37 | Hurt |
| Parallelize full layers rather than only attention layers (ablation) | EasySpec · MMLU, Qwen2-72B-Instruct (1.5B drafter), LP size 2 | acceptance rate: 0.76 -> 0.68 | Hurt |
| Increase full-layer parallelization from LP size 2 to 3 (ablation) | EasySpec · MMLU, Qwen2-72B-Instruct (1.5B drafter) | token throughput: 27.37 -> 24.10 | Hurt |
| Increase full-layer parallelization from LP size 2 to 3 (ablation) | EasySpec · MMLU, Qwen2-72B-Instruct (1.5B drafter) | acceptance rate: 0.68 -> 0.51 | Hurt |
| Increase layer-parallel size from 1; at excessively large size (e.g. 5), throughput declines (ablation) | EasySpec · Llama-3-70B-Instruct (8B drafter), MMLU | token throughput | Mixed |
| Add bonus calibration (ablation) | EasySpec · All tested tree widths and layer-parallel sizes | acceptance rate | Helped |
| Vary tree attention width (ablation) | EasySpec · Llama-3-70B-Instruct (8B drafter), MMLU | token throughput | Mixed |
Surprising findings
- Parallelizing MLP layers as well as attention layers degrades throughput and acceptance, so only attention layers are parallelized.
- Tensor parallelism of the small draft models often slows them down (TP=1 is fastest for drafters), so multi-GPU resources sit idle during drafting.
Limitations
- Results depend on the specific 8xA100 hardware and inference system; speedups may vary on other platforms.
- Fuzzy approximation error can reduce acceptance rate (up to 7% drop), and the best layer-parallel size depends on tree width and model.
- Only tested with max 128 generated tokens and draft models from the same series; no training-based drafters compared beyond EAGLE-2.
Experimental setup
Datasets: MMLU HumanEval MATH IFEval MGSM Spec-Bench
Benchmarks: MMLU HumanEval MATH IFEval MGSM Spec-Bench
Metrics: token throughput acceleration ratio drafting time per 100 tokens verification time per 100 tokens acceptance rate throughput variance cosine similarity
Baselines: tensor-parallel decoding (TP) speculative decoding (+sd) tree attention (+tree) EAGLE-2
Builds on: speculative decoding tree attention tensor parallelism
Closest papers by meaning
- SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.92)
- Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.91)
- AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.89)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.89)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.89)
- DFlash: Block Diffusion for Flash Speculative Decoding (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.