Research map / Large language models / Language, safety and interpretability
Tandem Transformers for Inference Efficient LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tandem Transformers, pairing a small autoregressive model that attends to a large block-mode model's representations, and combines it with speculative decoding (SPEED) for faster inference with PaLM2 models.
Problem
Autoregressive generation in LLMs is sequential and slow, and existing speculative/parallel decoding drafters are either weak small models or fail to exploit the base LLM's representations.
Approach
A small autoregressive model generates blocks of gamma tokens while attending to the large model's (PaLM2-Bison) representations of prior tokens, which the large model processes in block mode; a linear projection aligns dimensions. Used standalone or as the SPEED drafter, with logit distillation and an adaptive block length router.
Key results
- Tandem-Distil (PaLM2-Bison + PaLM2-Gecko) reaches 58.61% accuracy on the pretraining set vs 56.50% for PaLM2-Gecko-Distil, and SuperGLUE 78.8 vs 62.8 for PaLM2-Gecko.
- Tandem is about 1.16x faster than PaLM2-Otter with comparable downstream performance (SuperGLUE 78.8 vs 79.0).
- Tandem-Distil + SPEED is about 1.11x to 1.17x faster than SPEED with distilled PaLM2-Gecko, and at least 2.19x faster than PaLM2-Bison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Give PaLM2-Gecko access to PaLM2-Bison representations in Tandem | PaLM2-Gecko · Ground-truth next-token prediction | accuracy: 55.06 -> 58.35 | Helped |
| Add logit distillation to Tandem-CE training (ablation) | Tandem-CE · Ground-truth next-token prediction | accuracy: 58.35 -> 58.61 | Helped |
| Use Tandem-Distil instead of PaLM2-Gecko-Distil as the SPEED drafter | SPEED with PaLM2-Bison verifier · Reddit, num-samples=1 | speedup over PaLM2-Bison: 2.169 \times -> 2.471 \times; 1.139 \times relative gain | Helped |
| Use Tandem-Distil instead of PaLM2-Gecko-Distil as the SPEED drafter | SPEED with PaLM2-Bison verifier · CNN/DailyMail, num-samples=1 | speedup over PaLM2-Bison: 2.219 \times -> 2.473 \times; 1.115 \times relative gain | Helped |
| Use Tandem-Distil instead of PaLM2-Gecko-Distil as the SPEED drafter | SPEED with PaLM2-Bison verifier · LM1B, num-samples=1 | speedup over PaLM2-Bison: 2.348 \times -> 2.610 \times; 1.112 \times relative gain | Helped |
| Use adaptive block length (ablation) | Tandem-Distil + SPEED · Reddit, num-samples=1 | speedup over Tandem-Distil + SPEED: 1.045 \times | Helped |
| Use adaptive block length (ablation) | Tandem-Distil + SPEED · CNN/DailyMail, num-samples=1 | speedup over Tandem-Distil + SPEED: 1.051 \times | Helped |
| Use adaptive block length (ablation) | Tandem-Distil + SPEED · LM1B, num-samples=1 | speedup over Tandem-Distil + SPEED: 1.093 \times | Helped |
| Replace PaLM2-Gecko with smaller PaLM2-XXXS as the Tandem secondary model (ablation) | Tandem-Distil + SPEED · LM1B, num-samples=1 | relative speedup over distilled secondary-model SPEED: 1.139 \times -> 1.243 \times | Helped |
Surprising findings
- Tandem with PaLM2-XXXS as secondary model improves latency further, with the gains over distilled XXXS-SPEED reaching 1.366x at num-samples 4 on LM1B.
Limitations
- Adaptive block length is hard to extend to larger num-samples or batch sizes, as stated by the authors.
- Experiments use proprietary PaLM2 models and an internal pretraining dataset, limiting reproducibility.
- Latency gains are reported only on TPUv5e and with batch size 1 for the main results.
Experimental setup
Datasets: PaLM2 pretraining dataset
Benchmarks: SuperGLUE TydiQA-GoldP Generative-tasks (SQuADv2, Natural Questions, TriviaQA, WebQuestions, Lambada) MBPP WMT22 CNN/DailyMail Reddit Posts LM1B
Metrics: accuracy F1 exact match cross entropy relative total variation distance end-to-end latency speedup acceptance rate
Baselines: PaLM2-Gecko PaLM2-Gecko-Distil PaLM2-Otter PaLM2-Bison PaLM2-XXXS-Distil SPEED with distilled PaLM2-Gecko
Builds on: speculative decoding logit distillation LoRA Medusa DistilSpec
Closest papers by meaning
- Multi-Branch Self-Drafting for LLM Inference Acceleration (AAAI 2025 · similarity 0.91)
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.90)
- Optimized Multi-Token Joint Decoding With Auxiliary Model for LLM Inference (ICLR 2025 · similarity 0.90)
- PARD: Accelerating LLM Inference with Low‑Cost PARallel Draft Model Adaptation (ICLR 2026 · similarity 0.90)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.89)
- Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.