Research map / Learning theory and algorithms / Optimal transport
SpecTr: Fast Speculative Decoding via Optimal Transport
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, Felix Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SpecTr K=8 instead of speculative K=1 (L=4) | speculative decoding · LM1B PaLM-2 | block efficiency: 2.4 -> 3.1 | Improved |
| use SpecTr K=8 instead of speculative K=1 (L=4) | speculative decoding · LM1B PaLM-2 | wall clock speedup: 1.67 -> 2.08 | Improved |
| use SpecTr K=8 instead of speculative K=1 (L=8) | speculative decoding · LM1B PaLM-2 | block efficiency: 2.9 -> 4.0 | Improved |
| use SpecTr K=8 instead of speculative K=1 (L=8) | speculative decoding · LM1B PaLM-2 | wall clock speedup: 1.56 -> 2.13 | Improved |
| increase draft length L from 4 to 8 (ablation) | speculative decoding K=1 · LM1B PaLM-2 | wall clock speedup: 1.67 -> 1.56 | Hurt |
| increase draft length L from 4 to 8 (ablation) | SpecTr K=8 · LM1B PaLM-2 | wall clock speedup: 2.08 -> 2.13 | Improved |
| increase number of drafts K from 1 to 8 (L=4) (ablation) | SpecTr, 2M draft · LM1B 97M large | block efficiency: 1.86 -> 2.56 | Improved |
| increase number of drafts K from 1 to 8 (L=8) (ablation) | SpecTr, 2M draft · LM1B 97M large | block efficiency: 1.91 -> 2.68 | Improved |
| increase batch from 1 to 8 for large model scoring (length 4) (ablation) | 97M transformer · GPU latency | relative latency: 1.00 -> 1.10 | No clear effect |
| increase batch from 1 to 8 for large model scoring (length 8) (ablation) | 97M transformer · GPU latency | relative latency: 1.01 -> 1.42 | Mixed |
| increase batch from 1 to 8 for 6M draft (length 4) (ablation) | 6M transformer · GPU latency | relative latency: 0.18 -> 0.20 | No clear effect |
| wall clock vs theoretical block efficiency | SpecTr K=8 L=8 · LM1B PaLM-2 | speedup: 4.0 -> 2.13 | Hurt |
Closest papers by meaning
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.92)
- Speculative Speculative Decoding (ICLR 2026 · similarity 0.92)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- DFlash: Block Diffusion for Flash Speculative Decoding (ICML 2026 · similarity 0.91)
- Global Resolution: Optimal Multi-Draft Speculative Sampling via Convex Optimization (ICLR 2026 · similarity 0.91)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
