Research map / Large language models / Reasoning and chain of thought

SEED: Self-Speculative Decoding via Implicit Encoder–Decoder

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SEED, a self-speculative decoding method that views a decoder-only LLM as an implicit encoder-decoder, reusing verifier-cached deep KV states so a 2-layer decoder drafts tokens, trained with an added block-attention speculative loss.

Problem

Self-speculative decoding trades draft quality against drafting cost: early-exit drafts are cheap but shallow, while MTP drafts are deep but require a full forward pass per draft step.

Approach

Partition the transformer into a deep encoder (first l' layers) and a thin decoder (last layers); verification runs the full model and caches deep KV states, and a thin decoder drafts multiple tokens from raw embeddings cross-attending to that cache, trained with a block-causal speculative drafting loss added to standard SFT.

Key results

• Average speedup of 2.6x on Qwen3-1.7B-Base and 2.7x on Qwen3-4B-Base across four benchmarks, outperforming self-speculative baselines.
• On Qwen3-4B-Base, SEED reaches 76.0% GSM8K accuracy (vs 75.1% AR) and 74.9 tokens/s GSM8K throughput vs EAGLE-3's 64.4.
• Average throughput 43.3 tokens/s on CNN/Daily Mail for the 4B model is second to EAGLE-3 (49.9), while SEED is strongest on other tasks.

Surprising findings

• SEED improves task accuracy over standard AR fine-tuning on GSM8K, KodCode, and ScienceQA while accelerating decoding.
• Layer-skipping baseline LayerSkip shows substantial accuracy degradation (e.g., 62.2% vs 75.1% AR on GSM8K for 4B).

Limitations

• Only evaluated on Qwen3 1.7B and 4B base models; not validated at 8B or larger.
• Task-specific SFT setting only; not tested in general instruction tuning or pretraining.
• Fixed partition (top two layers as decoder) is not adaptive and may not be optimal across settings.

Experimental setup

Datasets: GSM8K KodCode ScienceQA CNN/Daily Mail

Benchmarks: GSM8K KodCode ScienceQA CNN/Daily Mail

Metrics: pass@1 accuracy decoding throughput (tokens/s) ROUGE-1 ROUGE-2 ROUGE-L speedup

Baselines: AR E2D2 Apple MTP EAGLE-3 PARD LayerSkip SWIFT DEL

Builds on: LayerSkip E2D2 Apple MTP EAGLE-3 speculative decoding

Closest papers by meaning

• Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.91)
• Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.91)
• SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
• SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification (ICML 2026 · similarity 0.90)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.90)
• Falcon: Faster and Parallel Inference of Large Language Models Through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree (AAAI 2025 · similarity 0.90)