Research map / Large language models / Reasoning and chain of thought

Attention-Level Speculation

ICML 2025 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes attention-level speculative parallelism (ALSpec), which approximates self-attention with attention sink, verifies it against exact attention via an L2 threshold, and overlaps subsequent ops to cut decode latency on multi-device inference.

Problem

Decode-stage latency of LLM inference at long context lengths is dominated by self-attention, and tensor and data parallelism show diminishing returns when scaled across devices.

Approach

Speculate on each layer's self-attention output using a cheap attention-sink approximation, run the post-attention ops speculatively on a separate device while exact attention is computed, and verify with a relative L2 threshold to commit or discard. A speculative flash decode kernel and a static-graph dynamic-concurrency runtime with priority gating make this practical.

Key results

• At 128K context with 87.5% speculation hit rate, ALSpec reduces attention latency overhead by up to 5x and gives up to 1.65x per-user throughput over full tensor parallelism on Tenstorrent N150 devices.
• With lambda=0.10, ALSpec matches or exceeds baseline scores on most evaluated tasks while speculating successfully on 18% to 90% of attention ops depending on task.
• Projected on H100s at 128K with 65% hit rate, ALSpec gives an estimated 28.5% tokens/s improvement vs 9.2% for full TP scaling from 4 to 8 GPUs.

Measured effects reported in the paper

Surprising findings

• Static attention-sink approximation degrades math and reasoning tasks (e.g., MMLU-Pro CoT), while dynamic verification recovers baseline accuracy.
• Full tensor parallelism gives almost no gain from 4 to 8 devices at long context lengths on N150 hardware.

Limitations

• Speedup numbers are estimates from device kernel durations that exclude host and dispatch latency.
• GPU results are projections; no CUDA implementation was built.
• Speculation hit rates vary widely by task (18% to 90%), and correctness was simulated by running both attention paths.

Experimental setup

Datasets: IFEval GPQA SWDE FDA GSM8K MATH MGSM MMLU-Pro HotpotQA RepoBench-P synthetic needle-in-a-haystack

Benchmarks: IFEval GPQA CoT SWDE FDA GSM8K CoT MATH MGSM CoT MMLU-Pro CoT HotpotQA RepoBench-P needle-in-a-haystack

Metrics: task score speculation hit rate decode latency per token tokens/s/user relative score to baseline

Baselines: full tensor parallelism attention sink attention sink at fixed layers data parallelism

Builds on: speculative decoding attention sink flash decoding Lipschitz continuity analysis of transformers

Closest papers by meaning

• Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference (ICLR 2025 · similarity 0.89)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.89)
• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.89)
• Mixture of Attentions For Speculative Decoding (ICLR 2025 · similarity 0.88)
• ATTS: Asynchronous Test-Time Scaling via Conformal Prediction (ICLR 2026 · similarity 0.88)
• Attention Drift: What Auto-Regressive Speculative Decoding Models Learn (NeurIPS 2026 · similarity 0.88)