Research map / Architectures and efficiency / Attention and transformers

UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniPrefill, a block-wise top-p token-dropping method applied at full-attention layers that propagates sparsity to all subsequent sublayers, speeding up long-context prefill in vLLM across full-attention and hybrid LLMs.

Problem

Prefill acceleration for long-context LLMs is largely limited to sparse attention, which helps little on hybrid architectures (linear or sliding-window plus full attention) and is hard to integrate with continuous batching in production engines like vLLM.

Approach

Estimate token importance from last-n query attention at full-attention layers, keep the minimal top-p block set, and drop the rest from all downstream sublayers (attention, linear/SWA, FFN), so savings cover GEMM as well as attention FLOPs; implemented as fused kernels with vLLM scheduler integration.

Key results

• On RULER at 128K, UniPrefill reaches 2.26x TTFT speedup on Llama-3.1-8B-Instruct with RULER average 90.45 vs. baseline 90.36.
• On Qwen3-Next-80B-A3B at 128K, UniPrefill achieves 1.68x TTFT speedup with average score 93.94 vs. baseline 94.76.
• In vLLM throughput tests, UniPrefill gains up to +109% (Llama-3.1-8B), +68% (Qwen3-Next-80B-A3B), and +42% (Gemma-3-12B) over standard prefill.

Surprising findings

• Qwen3-Next-80B-A3B shows throughput losses of 3-5% at batch size 1 for short contexts (4K-8K) under UniPrefill in vLLM.
• The ablation on random seeds reports identical RULER scores across seeds 0, 321, and 3467, which is unusual for a deterministic-free pipeline and was not explained.

Limitations

• Accuracy drops noticeably at some settings (e.g., Gemma-3-12B at 128K drops from 61.22 to 58.38; Qwen3-Next at 4K shows slightly lower throughput at BSZ 1).
• Evaluation limited to three models and a single benchmark (RULER).

Experimental setup

Datasets: RULER

Benchmarks: RULER

Metrics: RULER score TTFT speedup prefill throughput (tokens/s)

Baselines: LazyLLM SlimInfer MInference FlexPrefill XAttention ProxyAttn

Builds on: SnapKV vLLM MInference FlexPrefill

Closest papers by meaning

• FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.89)
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.89)
• Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.88)
• SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator (ICML 2025 · similarity 0.88)
• Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation (ICML 2025 · similarity 0.88)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.88)