Research map / Large language models / Reasoning and chain of thought
Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SubSpec, a training-free, lossless speculative decoding method for offloaded LLMs that builds a GPU-resident draft from 4-bit HQQ substitutes of offloaded layers, shared resident layers, and a shared KV-cache, reaching about 9-12.5x speedups on consumer GPUs.
Problem
Running LLMs larger than consumer GPU memory via parameter offloading is slow because of PCIe weight transfers on each forward pass; existing speculative decoding draft models either need same-family pretrained models or training and reach only modest acceptance lengths under offloading.
Approach
Build a highly aligned draft model without training by replacing the offloaded target layers with data-free 4-bit quantized substitutes kept on GPU, sharing the GPU-resident target layers and KV-cache, and sharpening draft probabilities (temperature 0.2) to counter cumulative-probability divergence in deep draft trees.
Key results
- SubSpec achieves 9.1x speedup on Qwen2.5 7B on MT-Bench under an 8GB VRAM limit (25.35 tokens/s, tau 27.08 at greedy decoding).
- Averages 12.5x speedup for Qwen2.5 32B across five benchmarks under a 24GB VRAM limit (6.50 tokens/s, tau 28.97).
- Outperforms EAGLE-2 and Qwen2.5 1.5B draft by roughly 30% to 50% in throughput on the same target without additional training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SubSpec instead of Q 1.5B as the draft model | Qwen2.5-7B speculative decoding, temperature 0 · Mean (five benchmarks) | tokens/s: 17.73 -> 27.09 | Helped |
| Use SubSpec instead of Q 1.5B as the draft model | Qwen2.5-14B speculative decoding, temperature 0 · Mean (five benchmarks) | tokens/s: 9.56 -> 12.56 | Helped |
| Use SubSpec instead of Q 1.5B as the draft model | Qwen2.5-32B speculative decoding, temperature 0 · Mean (five benchmarks) | tokens/s: 5.09 -> 6.50 | Helped |
| Use SubSpec instead of Q 1.5B as the draft model | Qwen2.5-7B speculative decoding, temperature 0.6 · Mean (five benchmarks) | tokens/s: 14.71 -> 18.70 | Helped |
| Use SubSpec instead of Q 1.5B as the draft model | Qwen2.5-14B speculative decoding, temperature 0.6 · Mean (five benchmarks) | tokens/s: 7.50 -> 7.83 | Helped |
| Use SubSpec instead of Q 1.5B as the draft model | Qwen2.5-32B speculative decoding, temperature 0.6 · Mean (five benchmarks) | tokens/s: 3.82 -> 4.06 | Helped |
| Add shared KV-Cache (ablation) | SubSpec on Qwen2.5-7B · MT-Bench, temperature 0, 8GB | tokens/s: 19.54 -> 21.99 | Helped |
| Add draft probability sharpening (ablation) | SubSpec on Qwen2.5-7B · MT-Bench, temperature 0, 8GB | tokens/s: 21.99 -> 23.66 | Helped |
| Add asynchronous data transfer (ablation) | SubSpec on Qwen2.5-7B · MT-Bench, temperature 0, 8GB | tokens/s: 23.66 -> 25.35 | Helped |
| Lower SubSpec draft temperature from 1.0 to 0.2 (ablation) | SubSpec with Qwen2.5-7B target · MT-Bench, greedy decoding | average acceptance length (τ): 25.29 -> 27.08 | Helped |
Surprising findings
- Lowering draft temperature to 0.2 raises acceptance length even for the target model used as its own draft (27.79 to 34.50), revealing cumulative probability divergence in greedy tree decoding.
- A standalone 4-bit HQQ model reaches 135.84 tokens/s, faster than SubSpec's 24.29 tokens/s, though it alters outputs.
Limitations
- Requires at least about 7.1GB GPU memory for Qwen2.5 7B, so fewer target layers stay resident than in other SD methods.
- Only 4-bit substitute quantization was explored; 2/3-bit and other alignment trade-offs are not studied.
- Mainly suited to dense architectures; MoE models need further adaptation.
Experimental setup
Datasets: MT-Bench HumanEval GSM8K Alpaca CNN/Daily Mail AIME 2024 MATH 500 GPQA Diamond LiveCodeBench
Benchmarks: MT-Bench HumanEval GSM8K Alpaca CNN/Daily Mail AIME 2024 MATH 500 GPQA Diamond LiveCodeBench
Metrics: tokens/s average acceptance length speedup ratio
Baselines: EAGLE-2 Qwen2.5 1.5B draft Llama3.2 1B draft SpecExec Dovetail GPTQ AWQ HQQ vanilla offloading
Builds on: EAGLE-2 SpecExec HQQ Sarathi-Serve Spec-Bench
Closest papers by meaning
- SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.93)
- EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization (NeurIPS 2025 · similarity 0.91)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.90)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.90)
- 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.90)
- Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.