Research map / Architectures and efficiency / Attention and transformers
Block-Attention for Efficient Prefilling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Block-attention, which encodes each retrieved passage as an independent block with cached KV states and position re-encoding, then block-fine-tunes an LLM so RAG accuracy matches full attention while cutting time-to-first-token sharply.
Problem
RAG inputs contain many retrieved passages, making prefilling slow; KV caches of passages cannot be reused because autoregressive KV states depend on preceding context.
Approach
Use block-diagonal attention so each passage block attends only within itself while the final query block attends to all previous blocks; re-encode RoPE positions of cached blocks, and fine-tune the LLM with this mask (alongside full attention) so it adapts to block mode and can switch between modes without loss.
Key results
- Block-attention model Tulu3-block-ft matches Tulu3-RAG within about 1% on four RAG benchmarks (2Wiki 72.2 vs 73.2, HQA 72.3 vs 74.8, NQ 60.4 vs 61.5, TQA 75.1 vs 75.8).
- Without block fine-tuning, average RAG accuracy falls from 66.1% to 49.9% (Llama-3.1-Tulu-3-8B-SFT based), and Tulu3-block-w/o-ft drops 16.2% on average versus Tulu3-RAG.
- At a 32K total prompt length with a 50-token user input, TTFT drops from 3638 ms to 45 ms (98.7% reduction) and FLOPs-TFT are reduced by 99.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the model for Block-attention (ablation) | Tulu3-RAG · RAG benchmarks (2Wiki, HQA, NQ, TQA) | accuracy: 42.9, 42.1, 48.3, 66.5 -> 72.2, 72.3, 60.4, 75.1 | Helped |
| Add position re-encoding (ablation) | Tulu3-block-ft · RAG benchmarks (2Wiki, HQA, NQ, TQA) | accuracy: 68.9, 69.9, 59.2, 74.4 -> 72.2, 72.3, 60.4, 75.1 | Helped |
| Switch Block-attention to full-attention mode (ablation) | Tulu3-block-ft · RAG benchmarks (2Wiki, HQA, NQ, TQA) | accuracy: 72.2, 72.3, 60.4, 75.1 -> 73.6, 75.2, 62.2, 76.2 | No clear effect |
| Use Block-attention instead of full-attention with block fine-tuning | Tulu3-RAG · RAG benchmarks (2Wiki, HQA, NQ, TQA) | accuracy: 73.2, 74.8, 61.5, 75.8 -> 72.2, 72.3, 60.4, 75.1 | No clear effect |
| Apply Block-attention with cached passage KV states | Llama3-block-ft · 32K prompt | TTFT: 3638 -> 45; 98.7% | Helped |
| Apply Block-attention with cached passage KV states | Llama3-block-ft · 32K prompt | FLOPs-TFT: 4.9e+14 -> 7.5e+11; 99.8% | Helped |
| Apply Block-attention with cached passage KV states | Llama3-block-ft · 512 prompt | TTFT: +48% | Helped |
| Apply Block-attention with cached passage KV states | Llama3-block-ft · 512 prompt | FLOPs-TFT: 90.1% | Helped |
| Use Block-attention in ICL tasks after fine-tuning | Tulu3-block-ft · GSM8K, MATH, BBH, DROP | accuracy: 75.5, 29.2, 68.5, 9.4 -> 75.7, 28.8, 65.3, 14.4 | Mixed |
| Switch Block-attention model to full-attention in zero-shot tasks (ablation) | Tulu3-block-ft · IFEval, HumanEval, MMLU | accuracy: 68.5, 58.5, 63.7 -> 70.0, 59.1, 63.0 | No clear effect |
Surprising findings
- Naive block attention without fine-tuning causes severe accuracy degradation, while the same model fine-tuned with block attention recovers full accuracy and can also switch to full attention without loss.
- Training-free parallel prompt encoding methods (PromptCache, Superposition) show severe degeneration, with accuracy often far below the full-attention baseline.
Limitations
- Evaluation is limited to RAG, ICL, and general benchmarks; other scenarios such as code and multi-turn dialogue are left for future work.
- Training-free block attention degrades accuracy heavily, so the method requires fine-tuning the model.
- Game AI results are from an unreleased internal game and cannot be verified; efficiency measurements omit KV cache cost.
Experimental setup
Datasets: Tulu3-SFT mixture TriviaQA 2WikiMultiHopQA
Benchmarks: Natural Questions TriviaQA HotpotQA 2WikiMultiHopQA NarrativeQA MMLU BBH DROP MATH GSM8K HumanEval IFEval
Metrics: accuracy TTFT FLOPs-TFT
Baselines: PromptCache Superposition (parallel prompt encoding) Tulu3-SFT Tulu3-RAG Block-attention without fine-tuning
Builds on: RoPE Flash-Attention Tulu 3
Closest papers by meaning
- Block Sparse Flash Attention (NeurIPS 2026 · similarity 0.86)
- Retrieval from Within: An Intrinsic Capability of Attention-Based Models (NeurIPS 2026 · similarity 0.86)
- MoBA: Mixture of Block Attention for Long-Context LLMs (NeurIPS 2025 · similarity 0.85)
- MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLM (NeurIPS 2026 · similarity 0.85)
- Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.85)
- CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.