Research map / Architectures and efficiency / RNNs and normalization
It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add retention gate (ablation) | Yaad · Average LM score (Yaad ablation) | avg LM score: 50.63 -> 53.98 | Improved |
| make Huber threshold delta input-dependent (ablation) | Yaad · Average LM score (Yaad ablation) | avg LM score: 52.19 -> 53.98 | Improved |
| use Huber loss instead of pure l2 (ablation) | Yaad · Average LM score (Yaad ablation) | avg LM score: 52.86 -> 53.98 | Improved |
| use Huber loss instead of pure l1 (ablation) | Yaad · Average LM score (Yaad ablation) | avg LM score: 53.04 -> 53.98 | Improved |
| use 2-layer MLP memory instead of linear memory (ablation) | Yaad · Average LM score (Yaad ablation) | avg LM score: 51.57 -> 53.98 | Improved |
| replace Gated DeltaNet with Moneta | 1.3B model · WikiText | perplexity: 16.42 -> 15.52 | Improved |
| replace Gated DeltaNet with Yaad | 1.3B model · WikiText | perplexity: 16.42 -> 15.18 | Improved |
| use Moneta instead of Gated DeltaNet | S-NIAH · S-NIAH average | accuracy: 75.8 -> 93.5 | Improved |
| use Memora instead of Gated DeltaNet | 760M model · WikiText | perplexity: 21.18 -> 22.28 | Hurt |
| vary p of lp attentional bias (p=4 vs p=3) (ablation) | Moneta · LM across 2K-16K context | performance | Mixed |
Closest papers by meaning
- Titans: Learning to Memorize at Test Time (NeurIPS 2025 · similarity 0.86)
- Understanding the Differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks (NeurIPS 2024 · similarity 0.86)
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.86)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.85)
- Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.85)
- Zoology: Measuring and Improving Recall in Efficient Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
