Research map / Architectures and efficiency / Attention and transformers
Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Twilight adds adaptive, top-p (nucleus-style) budget pruning as a hierarchical second stage on top of existing top-k sparse attention methods, cutting attended tokens by up to 98% with little accuracy loss.
Problem
Top-k sparse attention for long-context LLMs uses a fixed token budget, causing over-selection for focused attention and under-selection for diffuse attention, and optimal budgets vary by layer, head, prompt and algorithm.
Approach
Apply top-p (nucleus) thresholding to the attention weights of tokens pre-selected by a base sparse attention algorithm (select-then-prune), so the budget adapts per head and query; uses 4-bit quantized key cache for estimating weights and binary-search top-p kernels.
Key results
- Twilight prunes up to 98% of tokens over base top-k budgets on Longbench, raising Longchat-7B-v1.5-32k average from 36.78 (full) to 38.52 with Twilight on DS-based selection (38.71).
- Quest-Twi achieves about 1.4x self-attention speedup over Quest, and up to 15.8x over FlashAttention2; end-to-end decoding up to 1.35x over Quest.
- On LLaMA-3.1-8B-Instruct Longbench, Twilight with DS stays within 1% of full attention (51.73 vs 52.01 for full attention, with Twilight average budget 446 vs 128k context).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Twilight pruning to Quest | Quest · Longbench (12-task average), Longchat-7B-v1.5-32k | average score: 37.33 -> 38.04 | Helped |
| Apply Twilight pruning to DS | DS · Longbench (12-task average), Longchat-7B-v1.5-32k | average score: 36.62 -> 38.71 | Helped |
| Apply Twilight pruning to full attention | Full attention · Longbench (12-task average), LLaMA-3.1-8B-Instruct | average score: 52.01 -> 51.64 | Hurt |
| Apply Twilight pruning to Quest | Quest · RULER average, LLaMA-3.1-8B-Instruct | average score: 81.62 -> 86.65 | Helped |
| Apply Twilight pruning to DS | DS · RULER average, LLaMA-3.1-8B-Instruct | average score: 87.18 -> 87.88 | Helped |
| Use Twilight pruning instead of the fixed-budget baseline | LLaMA-2-7B-Chat · PG-19 | perplexity: Full 7.503; Twilight 7.600; Quest 14.15; DS 7.622 | Mixed |
| Use Twilight pruning instead of the fixed-budget baseline | LLaMA-3.1-8B-Instruct · GSM8K flexible | accuracy: Full 0.7726; Twilight 0.7771 | Helped |
| Use Twilight pruning instead of the fixed-budget baseline | LLaMA-3.1-8B-Instruct · COQA exact match | exact match: Full 0.6363; Twilight 0.6325 | Hurt |
| Apply Twilight to Quest | Quest · Self-attention operator | speedup: 1.4× | Helped |
| Apply Twilight to Quest | Quest · End-to-end decoding | speedup: 1.35× | Helped |
| Use a top-p threshold around 0.85 (ablation) | Twilight on Longchat-7B-v1.5-32k · PG-19 and TrivialQA | accuracy and sparse attention latency | Mixed |
| Use 4-bit rather than 2-bit key-cache quantization (ablation) | Twilight top-p weight estimation · Attention-weight estimation | stability of cumulative attention weights | Helped |
Surprising findings
- Quest with a fixed budget of 256 drops to 31.26 on Longchat, far below full attention, while the same base method with Twilight reaches 38.04.
- Plain top-k replacement of Quest's top-k with top-p does not work due to Quest's per-page max pooling layout, requiring a hierarchical redesign.
Limitations
- Head-wise dynamic budgets become group-wise under GQA, reducing adaptivity.
- Estimation overhead of the top-p pruner limits gains when loading tokens is not the bottleneck.
- Additional INT4 key cache adds memory overhead (about 1/8 extra KV cache) unless the base algorithm already stores INT4 keys.
Experimental setup
Datasets: Longbench RULER GSM8K COQA PG-19 Qasper GovReport LCC
Benchmarks: Longbench RULER GSM8K COQA PG-19
Metrics: score perplexity exact match flexible/strict accuracy latency speedup TPOT
Baselines: Quest Double Sparsity (DS) MagicPIG FlashAttention2 FlashInfer StreamingLLM SnapKV
Builds on: Quest Double Sparsity FlashInfer nucleus sampling
Closest papers by meaning
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.89)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.89)
- Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models (NeurIPS 2026 · similarity 0.89)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.89)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.88)
- Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.