Research map / Large language models / Reasoning and chain of thought

Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ESP, a training-free multi-token prediction method that probes a frozen LLM with mask tokens drawn from its embedding space, builds a dynamic speculative token tree, and verifies losslessly to speed decoding.

Problem

Speculative decoding and multi-token prediction usually require training auxiliary heads, modifying weights, or external draft models, which is costly and unsuitable for edge devices. The paper asks whether frozen LLMs can predict multiple future tokens in parallel without any training.

Approach

Inject on-the-fly mask tokens (initialized as the mean of prompt embeddings and updated with generated tokens) into the prompt with a custom tree attention mask, sample top-K candidates from their logits to form a dynamic draft tree under a block-complexity budget, and verify losslessly with the base model; a cosine-similarity lemma justifies why mask-token states can recover the next-next token.

Key results

• ESP improves acceptance length by 7-11% over LADE on LLaMA3 models and 7-8% on Qwen3 models (abstract claim).
• On LLaMA3.1-8B-Instruct at BC=60, ESP reaches tau=1.71 and 1.38x speedup over autoregressive decoding on H100.
• Efficient tree-attention caching gives roughly 21-22% average throughput gain over naive implementation.

Surprising findings

• Mean-of-prompt mask initialization outperformed hard last-K token initialization and Gaussian sampling from the embedding table.
• Increasing the number of mask tokens beyond two yields diminishing returns, since models are trained only for next-token prediction.

Limitations

• Speedups are modest (typically 1.2-1.4x) and are below the strongest baseline on coding, RAG and summarization where n-gram copying helps STAND.
• Theory relies on a cosine-similarity assumption and does not explain why mask-token alignment emerges across layers.
• Tree attention masks become varying per pass with multiple mask tokens, adding implementation overhead.

Experimental setup

Datasets: SpecBench Dolly-Databricks GSM8K

Benchmarks: SpecBench

Metrics: average acceptance length (tau) speedup ratio (S/R) block efficiency reduction in model forward calls

Baselines: PLD STAND LADE

Builds on: Lookahead Decoding Prompt Lookup Decoding EAGLE-2 Medusa

Closest papers by meaning

• Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.91)
• L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models (NeurIPS 2025 · similarity 0.90)
• Optimized Multi-Token Joint Decoding With Auxiliary Model for LLM Inference (ICLR 2025 · similarity 0.90)
• OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference (ICML 2025 · similarity 0.88)
• Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding (ICML 2025 · similarity 0.88)
• Multi-Token Residual Prediction (NeurIPS 2026 · similarity 0.88)