Research map / Architectures and efficiency / Attention and transformers
Memory Caching: RNNs with Growing Memory
Ali Behrouz, Zeman Li, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, Vahab Mirrokni
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Gated Residual Memory caching | Titans 760M · Commonsense reasoning average | average accuracy: 51.56 -> 52.55 | Improved |
| add Gated Residual Memory caching | DLA 1.3B · Commonsense reasoning average | average accuracy: 53.72 -> 55.96 | Improved |
| add Gated Residual Memory caching | Titans · In-context retrieval average | average accuracy: 31.75 -> 40.50 | Improved |
| add GRM caching | DLA · S-NIAH-2 16K | accuracy: 28.2 -> 54.8 | Improved |
| use fixed-size segmentation instead of Log-Linear++ logarithmic segmentation (ablation) | DLA · S-NIAH-2 16K | accuracy: 18.0 -> 54.8 | Improved |
| add Log-Linear++ segmentation | Titans 760M · Commonsense reasoning average | average accuracy: 51.56 -> 51.74 | No clear effect |
| make gamma depend on segment context, not just input (ablation) | Titans (GRM) · Retrieval ablation | accuracy: 33.0 -> 40.5 | Improved |
| add input-dependent gating (ablation) | Titans (GRM) · Retrieval ablation | accuracy: 32.4 -> 40.5 | Improved |
| use deep memory instead of linear memory (ablation) | Titans (GRM) · Retrieval ablation | accuracy: 34.5 -> 40.5 | Improved |
| use separate u and q projections instead of shared (ablation) | Titans (GRM) · Retrieval ablation | accuracy: 36.6 -> 40.5 | Improved |
| vary segment size from 8 to 128 (ablation) | Memory Caching language modeling · Language modeling perplexity | perplexity: 16.38, 14.91, 15.26, 17.09 for 8, 32, 64, 128 | Mixed |
| use checkpoints of one memory instead of independent compressors (ablation) | Titans (GRM) · Language modeling and reasoning | accuracy: 57.2 -> 58.3 | Improved |
Closest papers by meaning
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.90)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.88)
- Quantifying Memory Utilization with Effective State-Size (ICML 2025 · similarity 0.88)
- Training Free Exponential Context Extension via Cascading KV Cache (ICLR 2025 · similarity 0.88)
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.88)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
