Research map / Large language models / Reasoning and chain of thought

Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation

ICML 2025 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpecPrefill, a training-free method where a small Llama-3.1-8B speculator scores prompt token importance via attention so only a selected subset is prefilled by a large Llama model, cutting TTFT and raising maximal QPS.

Problem

Time-to-first-token (TTFT) in LLM serving is compute-bound during prefill, with MLP cost dominating for large batches of short-to-medium prompts; existing prefill optimizations either need post-training or scale poorly.

Approach

Use a cheaper same-family model's attention (last-token attention aggregated over layers and heads, with look-ahead decoding, max-mean aggregation, and chunk-level selection) to pick important prompt tokens, then run the big model only on those tokens with their original position IDs preserved.

Key results

• On LongBench with Llama-3.1-405B, SpecPrefill achieves up to 7x maximal end-to-end QPS on Multi-Doc QA while keeping over 95% accuracy.
• Measured TTFT improvement reaches 7.66x for the 405B model at 32K sequence length, close to the 7.72x analytical estimate.
• For Llama-3.1-70B on LongBench, SpecPrefill with 10% keep rate reaches 52.74 average vs 53.55 baseline, outperforming RAG-Llama and LLMLingua at matched rates.

Measured effects reported in the paper

Surprising findings

• Dropping noisy tokens can improve accuracy over the full prompt on some tasks, e.g., code completion for 70B and MBPP/GPQA on standard tasks.
• Aggregation tasks (RULER CWE/FWE) degrade sharply under token dropping because every word matters.

Limitations

• Token dropping does not support explicit logits for all input tokens, so evaluation focuses on generative tasks.
• Information-dense short prompts (standard tasks without few-shot) are poorly compressible, limiting gains.
• Multi-turn conversations may fail without KV recomputation; keep rate is fixed per task rather than adaptive.

Experimental setup

Datasets: LongBench RULER MMLU IFEval GSM8K HumanEval MBPP ARC Challenge GPQA

Benchmarks: LongBench RULER Generative MMLU IFEval GSM8K 8-shot HumanEval MBPP ARC Challenge GPQA 8-shot

Metrics: task score TTFT speedup maximal QPS average query latency relative FLOPS overhead

Baselines: Llama-3.1-70B-Instruct Llama-3.1-405B-Instruct-FP8 RAG-Llama LLMLingua MInference

Builds on: speculative decoding vLLM GemFilter LLMLingua MInference

Closest papers by meaning

• SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding (ICML 2026 · similarity 0.89)
• SpecExit: Accelerating Large Reasoning Model via Speculative Exit (ICML 2026 · similarity 0.88)
• UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification (NeurIPS 2026 · similarity 0.88)
• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.88)
• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.88)
• Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.88)