Research map / Architectures and efficiency / Attention and transformers
Random-Access Infinite Context Length for Transformers
Amirkeivan Mohtashami, Martin Jaggi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add landmark retrieval (local 250, k=2) in place of plain 360-token attention (ablation) | GPT-2-like transformer · PG19, eval 512 | perplexity: 16.76 -> 16.23 | Improved |
| add landmark retrieval (local 250, k=2) in place of plain 360-token attention (ablation) | GPT-2-like transformer · arXiv, eval 512 | perplexity: 4.31 -> 4.01 | Improved |
| restrict retrieved blocks to be the same across tokens (per head only) (ablation) | Landmark attention · PG19, eval 4096, k=4 | perplexity: 14.72 -> 14.95 | Hurt |
| restrict retrieved blocks to be the same across heads (per token only) (ablation) | Landmark attention · PG19, eval 4096, k=4 | perplexity: 14.72 -> 14.89 | Hurt |
| increase retrieved blocks k from 2 to 4 at full flexibility (ablation) | Landmark attention · PG19, eval 2048 | perplexity: 15.14 -> 14.92 | Improved |
| increase eval length from 2048 to 4096 at k=4 (ablation) | Landmark attention · PG19 | perplexity: 14.92 -> 14.72 | Improved |
| increase k from 2 to 4 with per-head-only retrieval (ablation) | Landmark attention · PG19, eval 2048 | perplexity: 15.48 -> 15.10 | Improved |
| increase memory blocks from 20 to 40 at k=4, local 250 (ablation) | Landmark attention · PG19, eval 2048 | perplexity: 15.02 -> 14.92 | Improved |
| increase local context from 250 to 350 at k=2, 40 blocks (ablation) | Landmark attention · PG19, eval 2048 | perplexity: 15.14 -> 15.07 | Improved |
| increase memory blocks from 40 to 80 at k=4, eval 4096 (ablation) | Landmark attention · PG19, eval 4096 | perplexity: 14.79 -> 14.72 | Improved |
| train with context miss token (CMT), no cutoff (ablation) | Landmark attention · PG19, eval 2048 | perplexity: 16.28 -> 16.38 | Hurt |
| drop retrievals via CMT cutoff 0.5 (ablation) | Landmark attention with CMT · PG19, eval 2048 | perplexity: 16.38 -> 16.43 | Mixed |
Closest papers by meaning
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.90)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.87)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.87)
- Effective Context in Transformers: An Analysis of Fragmentation and Tokenization (NeurIPS 2026 · similarity 0.87)
- Core Context Aware Transformers for Long Context Language Modeling (ICML 2025 · similarity 0.86)
- Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
