Research map / Architectures and efficiency / Attention and transformers
NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies
Zhiyang Chen, Daliang Xu, Yinyuan Zhang, Chenghua Wang, Mengwei Xu, Yun Ma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace full 128k LM head with dynamic <3k in-context vocabulary | EAGLE-3 · Llama-3.1-8B-Instruct | avg speed (tokens/s): 362.2 -> 430.0 | Improved |
| use NanoSpec dynamic vocabulary instead of full vocabulary | EAGLE-2 · Llama-3.2-1B-Instruct | avg speed (tokens/s): 700.9 -> 900.7 | Improved |
| use dynamic vocabulary instead of static FR-Spec pruning | NanoSpec · Llama-3.2-1B-Instruct | avg acceptance length: 2.70 -> 3.11 | Improved |
| prune vocabulary dynamically versus full vocabulary | NanoSpec · Llama-3.1-8B-Instruct, EAGLE-2 | avg acceptance length: 3.80 -> 3.59 | Hurt |
| shrink draft LM head to dynamic vocabulary | NanoSpec · Llama-3.1-8B-Instruct | draft LM head latency (ms): 2.330 -> 0.237 | Improved |
| add asynchronous weight gathering with dual CUDA streams (ablation) | NanoSpec · Llama-3.1-8B-Instruct | avg speed (tokens/s): 364.1 -> 392.7 | Improved |
| add decoding-time candidates (draft and verify tokens) to context-only vocabulary (ablation) | NanoSpec · Llama-3.1-8B-Instruct | avg speed (tokens/s): 313.3 -> 351.4 | Improved |
| combine prompt-context and decoding-derived vocabulary sources (ablation) | NanoSpec · Llama-3.1-8B-Instruct | avg speed (tokens/s): 351.4 -> 364.1 | Improved |
| increase window size W_max beyond 4096 up to 8192 (ablation) | NanoSpec · Llama-3.1-8B-Instruct | end-to-end speed | Hurt |
| increase W_max from 1024 to 2048 (ablation) | NanoSpec · Llama-3.1-8B-Instruct | avg acceptance length | Improved |
| increase W_max beyond about 2048 (ablation) | NanoSpec · Llama-3.1-8B-Instruct | avg acceptance length | No clear effect |
Closest papers by meaning
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.90)
- Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.90)
- MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding (ICLR 2025 · similarity 0.90)
- AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders (NeurIPS 2025 · similarity 0.89)
- SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
