Research map / Architectures and efficiency / Attention and transformers
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato, Elvis Nunez, Matthew Trager, Wei Xia, Stefano Soatto
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L2A conditional Global Attention instead of full attention (CLP) | Qwen2.5-7B, 128K training · Training throughput | tokens/GPU/s: 642.67 -> 1383.85 | Improved |
| Use L2A instead of CLP at inference prefill | Qwen2.5-7B · Time-to-first-token, 64K/128K | TTFT | Improved |
| Share QKVO projections between Local and Global Attention (ablation) | L2A, Qwen2.5-1.5B · HELMET average | score: 29.6 -> 27.6 at 128k | No clear effect |
| Share QKVO projections between Local and Global Attention (ablation) | L2A, Qwen2.5-1.5B · Global Attention sparsity | sparsity: up to 10% lower sparsity | Hurt |
| Learned context-dependent router instead of context-free random Bernoulli routing (ablation) | L2A, Qwen2.5-1.5B · Long-context aggregate | score: nearly 30% over strongest context-free baseline | Improved |
| Reduce sliding window size from 4K down to 0 (ablation) | L2A · Long-context performance | score | No clear effect |
| Reduce sliding window size from 4K down to 0 (ablation) | L2A · Sparsity and TTFT | sparsity, TTFT: SWA-0 about 2x higher TTFT than SWA-4K | Hurt |
| Increase sparsity regularizer lambda (ablation) | L2A · Long-context performance | score | Hurt |
| Prune Global Attention in layers with sparsity above 95% | L2A, Qwen2.5-1.5B · Long-context performance, KV cache | score, KV memory: 15 of 28 modules dropped, nearly 50% KV savings | Improved |
| Prune Global Attention in layers with sparsity above 95% | L2A, Qwen2.5-7B · Long-context performance | score: nearly 40% of layers pruned | No clear effect |
| Freeze FFN layers and train only token-mixing and LayerNorm (ablation) | L2A and baselines · Long-context training | score | Improved |
| Invoke Global Attention for all tokens with probability 0.1 during training (ablation) | L2A · Router training | router collapse | Improved |
Closest papers by meaning
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.92)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.90)
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.90)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.89)
- Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.89)
- DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
