Research map / Architectures and efficiency / Attention and transformers
Short Window Attention Enables Long-Term Memorization
Loïc Cabannes, Maximilian Beck, Maria Lomeli, Gergely Szilvasy, Matthijs Douze, Jade Copet, Pierre-Emmanuel Mazare, Gabriel Synnaeve, Herve Jegou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| shorten sliding window from 2048 to 128 in training | SWAX 1.4B · RULER NIAH (avg over tasks and lengths) | accuracy: outperforms 2048 window by 16 accuracy points; 88.9% higher recall | Improved |
| increase sliding window from 128 to 2048 | SWAX 1.4B · Code validation perplexity | val_PPL: 2.551 -> 2.523 | Improved |
| increase sliding window from 128 to 2048 | SWAX 1.4B · Short-context reasoning average | average accuracy: 39.81 -> 40.88 | Improved |
| add xLSTM layers to SWA (hybrid SWAX:128 vs pure SWA:128) | SWA 1.4B · Code validation perplexity | val_PPL: 3.036 -> 2.551 | Improved |
| add stochastic window size (128/2048) instead of fixed 2048 | SWAX 1.4B · Code validation perplexity | val_PPL: 2.523 -> 2.502 | Improved |
| add stochastic window size instead of fixed 128 | SWAX 1.4B · Short-context average | average accuracy: 39.81 -> 40.81 | Improved |
| add stochastic window size instead of fixed 2048 | SWAX 7B · Short-context average | average accuracy: 49.32 -> 49.52 | Improved |
| add stochastic window size instead of fixed 2048 | SWAX 1.4B · Short-context average | average accuracy: 40.88 -> 40.81 | No clear effect |
| add stochastic window size instead of fixed 2048 | SWAX 7B · RULER NIAH average | niah_average: 29.52 -> 37.73 | Improved |
| add stochastic window size instead of fixed 2048 | SWAX 1.4B · Babilong | average score: 7.29 -> 9.35 | Improved |
| add stochastic window size instead of fixed 2048 (Gated DeltaNet) | SWAX/GDN 1.4B · Ruler multiquery NIAH | accuracy: 28.80 -> 40.90 | Improved |
| add stochastic window size instead of fixed 2048 (Gated DeltaNet) | SWAX/GDN 1.4B · LongBench | average score: 11.42 -> 9.94 | Hurt |
Closest papers by meaning
- Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity (NeurIPS 2026 · similarity 0.86)
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.86)
- STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs (NeurIPS 2026 · similarity 0.86)
- Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access (NeurIPS 2025 · similarity 0.85)
- Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling (ICLR 2025 · similarity 0.85)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
