Research map / Large language models / Language, safety and interpretability
Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Superposed Decoding, which generates k drafts in one autoregressive pass by feeding a probability-weighted superposition of the drafts' token embeddings plus n-gram interpolation, requiring no retraining.
Problem
Producing k auto-complete drafts with standard decoding requires k separate autoregressive inference passes, making multi-draft suggestions computationally expensive.
Approach
Superpose (probability-weighted sum) the embeddings of the most recent tokens of the k drafts as a single LM input, expand to k^2 candidate drafts from top-k tokens, and rescore with n-gram interpolation (n=2..6) to keep the top k; relies on approximate linearity of LM representations and needs no training.
Key results
- Superposed Decoding's best-of-3 drafts have average perplexity 4.63 vs 5.17 for Nucleus Sampling on Llama-2-7B (about 5% lower).
- SPD is 2.44x faster than the next-fastest method at k=3 and 3.54x faster at k=8 (wall-clock, unoptimized code).
- In a compute-normalized human study, SPD drafts were preferred about 63.6% of the time over Nucleus Sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate 3 drafts with Superposed Decoding instead of one Nucleus Sampling draft | Llama-2-7B · OpenWebText test split, best draft perplexity | average perplexity: 5.17 -> 4.63 | Helped |
| Generate 3 drafts with Superposed Decoding instead of Nucleus Sampling | Llama-2-7B · OpenWebText, 3 drafts | speedup vs Nucleus Sampling: 2.44× faster | Helped |
| Add 3 Superposed Decoding drafts to each Nucleus Sampled generation | Nucleus Sampling · TriviaQA and Natural Questions, compute budgets through 100 | coverage: TriviaQA: 51.04 -> 51.82 (n=1), 76.56 -> 79.95 (n=100); Natural Questions: 14.32 -> 15.63 (n=1), 42.97 -> 45.57 (n=100) | Helped |
| Offer three Superposed Decoding drafts rather than one Nucleus Sampling draft | Llama-2-7B · OpenWebText human evaluation | human preference win rate: 63.6% | Helped |
| Use Mistral 7B as the underlying model for Superposed Decoding | Superposed Decoding · OpenWebText test split, best of 3 drafts | average perplexity: 11.42 -> 10.87 | Helped |
| Use n-grams 2–4 instead of n-grams 2–6 (ablation) | Superposed Decoding · OpenWebText test split, best of 3 drafts | average perplexity: 5.17 -> 5.18 | No clear effect |
| Increase the number of Superposed Decoding drafts to k≥3 (ablation) | Superposed Decoding · OpenWebText, average best perplexity | average perplexity | Helped |
| Reset superposed tokens every s timesteps (ablation) | Superposed Decoding · Long-form generation | generation quality | Helped |
| Reset superposed tokens during long-form generation (ablation) | Superposed Decoding · 200-token generations | repetition | Helped |
| Vary prefix length (ablation) | Superposed Decoding · OpenWebText, prefix lengths 5, 15, 25, 40 | average perplexity | No clear effect |
| Vary generation length (ablation) | Superposed Decoding · OpenWebText, generation lengths 5, 10, 15, 20 | average perplexity | No clear effect |
| Use shorter generations (ablation) | Superposed Decoding · Draft diversity across prefixes | Self-BLEU | Helped |
Surprising findings
- Individual SPD drafts are worse than Nucleus Sampling or Greedy Decoding per draft; the gain comes from the best-of-k aggregate.
- Superposed Decoding drafts are syntactically but not semantically diverse.
Limitations
- Quality is bounded by the n-gram models, which need about 14 GB of storage.
- Linearity of representations degrades after about 10 timesteps, limiting generation length to short drafts.
- Drafts are syntactically but not semantically diverse.
Experimental setup
Datasets: OpenWebText RedPajama (n-gram models, ~200M tokens from 200k documents) TriviaQA Natural Questions
Benchmarks: OpenWebText test perplexity TriviaQA Natural Questions Amazon Mechanical Turk human preference
Metrics: perplexity exact match precision P@k coverage wall-clock latency Self-BLEU unique n-gram proportion human win rate
Baselines: Nucleus Sampling Greedy Decoding Beam Search Top-k Sampling Multi-token prediction Medusa
Builds on: Nucleus Sampling Beam Search linear representation hypothesis infini-gram
Closest papers by meaning
- Multi-Branch Self-Drafting for LLM Inference Acceleration (AAAI 2025 · similarity 0.88)
- Accelerating Blockwise Parallel Language Models with Draft Refinement (NeurIPS 2024 · similarity 0.88)
- Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion (NeurIPS 2026 · similarity 0.87)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.87)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.87)
- XTC: Head-Aware Sampling by Excluding Top Choices (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.