Research map / Architectures and efficiency / Attention and transformers

Attention Drift: What Auto-Regressive Speculative Decoding Models Learn

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies "attention drift" in EAGLE-3 and MTP drafters, traces it to unnormalized residual magnitude growth across speculation steps, and proposes post-norm drafter states plus RMSNorm on target hidden states to improve acceptance length under template perturbation, long context, and standard benchmarks.

Problem

Speculative decoding drafters (EAGLE-3) degrade sharply under chat-template perturbation and long-context inputs; the causes of this fragility inside the drafter are unclear.

Approach

Attention drift (attention moving from the sink/prompt onto recently drafted tokens) is driven by hidden-state magnitude growth through the unnormalized pre-norm residual chain, so applying post-norm to drafter hidden states and RMSNorm to each captured target hidden state stabilizes drafting and improves robustness.

Key results

• Post-norm improves acceptance length over pre-norm EAGLE-3 by up to 2x under template perturbation (e.g., Qwen3-8B no template, no BoS 2.21 vs 1.64).
• On LongBench, post-norm outperforms pre-norm by 20-25% across categories (e.g., Gov Report with SWA+Sys 1.70 vs 0.47).
• Across seven SGLang benchmarks, post-norm gains up to 12% (e.g., GSM8K +10% on GPT-oss low reasoning).

Surprising findings

• Gated attention removes the attention sink and drift but still shows high template sensitivity, so removing the sink alone is insufficient.
• Combining gated attention with post-norm collapses attention entropy to about 0.62, creating a new pathology.

Limitations

• Focuses on EAGLE-3; MTP drift mechanism not fully investigated and fix not evaluated on other drafter designs.
• Experiments limited to models below 120B parameters due to compute.

Experimental setup

Datasets: Open-PerfectBlend Nemotron post-training dataset MT-Bench GSM8K HumanEval LongBench

Benchmarks: MT-Bench GSM8K Alpaca HumanEval MATH-500 GPQA LiveCodeBench LongBench multi-turn long-context chat

Metrics: acceptance length accuracy under template perturbation attention sink and self-token attention hidden-state RMS

Baselines: pre-norm EAGLE-3 gated attention drafter MTP heads

Builds on: EAGLE-3 Medusa pre-norm transformer gated attention SpecForge train-time-test

Closest papers by meaning

• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.90)
• SimSD: Simple Speculative Decoding in Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.89)
• SEED: Self-Speculative Decoding via Implicit Encoder–Decoder (NeurIPS 2026 · similarity 0.89)
• Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning (ICLR 2026 · similarity 0.89)
• SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.89)