Research map / Large language models / Language, safety and interpretability

AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism

ICML 2025 · Large language models › Language, safety and interpretability · approach: Parallel and early-exit decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaDecode, which trains lightweight intermediate-layer LM heads (KL-distilled, frozen backbone) to adaptively emit tokens early and runs deferred upper-layer KV computations in parallel, with a rejection-sampling verification step that preserves output parity with autoregressive decoding.

Problem

Autoregressive LLM decoding is sequential and slow for long outputs; existing speedups either need an auxiliary drafter, alter architecture or training, or (layer skipping) break output consistency due to missing KV cache at skipped layers.

Approach

Train small per-layer LM heads (a learned d×d transformation applied on the frozen final LM head) via KL distillation to the final-layer distribution; when an intermediate head is confident (p > γ), emit the token early, defer the remaining layers' KV computation and batch them in parallel with later tokens, and verify with modified rejection sampling so outputs match vanilla decoding.

Key results

• AdaDecode achieves up to 1.73x speedup on HumanEval with CodeLlama-34B-Instruct, versus 1.40x for the best baseline SWIFT-style and 1.41x for SpecDecode with 7B drafter.
• On Llama3.1-8B-Instruct, AdaDecode reaches 1.14x (XSum), 1.51x (HumanEval), 1.48x (GSM8K), above SpecDecode with a 1B drafter.
• Ablation on HumanEval: removing verification drops consistency ratio to 0.652; the lightweight heads use 48M new parameters versus 1.5B for full-parameterized heads, with comparable speedup.

Measured effects reported in the paper

Surprising findings

• Removing verification yields a slightly higher speedup (1.64x) but drastically lowers output consistency to 0.652, showing that unverified early predictions diverge substantially.
• Naively combining adaptive layer parallelism with tree-based speculative decoding did not outperform tree-based decoding alone.

Limitations

• May incur higher FLOPs when early-predicted tokens are rejected and must be reverted.
• Focuses only on decoding acceleration and inherits general LLM issues like hallucination and unsafe content.
• Heads are trained per task in-domain; mixed-domain heads are notably slower, and low-rank adaptation for fewer parameters is left to future work.

Experimental setup

Datasets: XSum MBPP GSM8K HumanEval

Benchmarks: XSum HumanEval GSM8K

Metrics: throughput (tokens/s) speedup vs vanilla decoding consistency ratio (string match) early prediction rate verification rejection rate

Baselines: SpecDecode LookAhead Self-SpecDecode SWIFT EESD LayerSkip Medusa EAGLE

Builds on: speculative decoding early exiting LayerSkip Medusa LookAhead decoding SWIFT

Closest papers by meaning

• Accelerating Diffusion LLMs via Adaptive Parallel Decoding (NeurIPS 2025 · similarity 0.92)
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (ICML 2024 · similarity 0.91)
• Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding (NeurIPS 2025 · similarity 0.91)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.91)
• SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.91)