Research map / Large language models / Language, safety and interpretability

Depth Exploration for LLM Decoding

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Parallel and early-exit decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Depth Exploration Decoding (DEX), a lossless autoregressive decoding method that explores multiple intermediate depths in parallel across GPUs and commits only final-depth-validated tokens, collapsing stale branches.

Problem

Lossless depth-adaptive LLM decoding selects a single non-final exit depth per token, which either exits too late (wasting layers) or too early (triggering full-depth fallback and discarding drafts), leaving speedup headroom.

Approach

Replace single exit-depth selection with parallel exploration over multiple candidate depths; each depth stage runs on its own explorer, the final-depth output is the commit reference, the shallowest matching branch is reused, and the exploration lattice is collapsed after each commit. Adds depth-coupled sampling to diversify proposals and lightweight residual adapters for standard LLMs.

Key results

• DEX outperforms LayerSkip, AdaDecode, and DEL in measured throughput on LS-CodeLlama-34B and LS-Llama-2-70B across GSM8K, HumanEval, and XSum.
• On Llama-2-70B, DEX-1/3 and DEX-1/4 are competitive with matched 3-GPU and 4-GPU PEARL, and DEX-1/8 achieves the highest throughput among evaluated configurations.
• Finer exploration resolution (more explorers) generally improves throughput, with exploration-based DEX beating a single-exit variant at finer resolutions.

Surprising findings

• Idealized depth-side speedup is bounded roughly by K (number of explorers) under uniform partition, so pure depth exploration has a resource-dependent ceiling.
• Standard LLMs' intermediate layers are poorly aligned with the LM head, so stable EADs concentrate near the final depth without adapters.

Limitations

• Pure depth exploration has a resource-dependent speedup bound of about K, requiring K GPUs for K explorers.
• Lattice expansion grows as 2^K-1 tokens per cycle, and gains may saturate at larger K.
• Combining depth exploration with token-axis speculative decoding is left to future work.

Experimental setup

Datasets: ShareGPT GSM8K HumanEval XSum

Benchmarks: GSM8K HumanEval XSum

Metrics: throughput (tokens/s) walltime speedup EAD-implied oracle throughput

Baselines: AR tensor parallelism LayerSkip self-speculative decoding AdaDecode DEL Lookahead PEARL EAGLE-2 EAGLE-3

Builds on: LayerSkip speculative decoding AdaDecode Gumbel-Max trick

Closest papers by meaning

• AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.89)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.87)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.87)
• Multi-Branch Self-Drafting for LLM Inference Acceleration (AAAI 2025 · similarity 0.86)
• Fast-dLLM++: Fr\'{e}chet Profile Decoding for Faster Diffusion LLM Inference (NeurIPS 2026 · similarity 0.86)
• DMax: Aggressive Parallel Decoding for dLLMs (NeurIPS 2026 · similarity 0.86)