AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Learning When to Attend: Conditional Memory Access for Long-Context LLMs

Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato, Elvis Nunez, Matthew Trager, Wei Xia, Stefano Soatto

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use L2A conditional Global Attention instead of full attention (CLP)Qwen2.5-7B, 128K training · Training throughputtokens/GPU/s: 642.67 -> 1383.85Improved
Use L2A instead of CLP at inference prefillQwen2.5-7B · Time-to-first-token, 64K/128KTTFTImproved
Share QKVO projections between Local and Global Attention (ablation)L2A, Qwen2.5-1.5B · HELMET averagescore: 29.6 -> 27.6 at 128kNo clear effect
Share QKVO projections between Local and Global Attention (ablation)L2A, Qwen2.5-1.5B · Global Attention sparsitysparsity: up to 10% lower sparsityHurt
Learned context-dependent router instead of context-free random Bernoulli routing (ablation)L2A, Qwen2.5-1.5B · Long-context aggregatescore: nearly 30% over strongest context-free baselineImproved
Reduce sliding window size from 4K down to 0 (ablation)L2A · Long-context performancescoreNo clear effect
Reduce sliding window size from 4K down to 0 (ablation)L2A · Sparsity and TTFTsparsity, TTFT: SWA-0 about 2x higher TTFT than SWA-4KHurt
Increase sparsity regularizer lambda (ablation)L2A · Long-context performancescoreHurt
Prune Global Attention in layers with sparsity above 95%L2A, Qwen2.5-1.5B · Long-context performance, KV cachescore, KV memory: 15 of 28 modules dropped, nearly 50% KV savingsImproved
Prune Global Attention in layers with sparsity above 95%L2A, Qwen2.5-7B · Long-context performancescore: nearly 40% of layers prunedNo clear effect
Freeze FFN layers and train only token-mixing and LayerNorm (ablation)L2A and baselines · Long-context trainingscoreImproved
Invoke Global Attention for all tokens with probability 0.1 during training (ablation)L2A · Router trainingrouter collapseImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy