Research map / Architectures and efficiency / Attention and transformers
Test-time regression: a unifying framework for designing sequence models with associative memory
Ke Alexander Wang, Jiaxin Shi, Emily B. Fox
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use unweighted recursive least squares instead of linear attention in MQAR with P=128 keys and d_model=64 | single test-time regression layer · MQAR (P=128, d_model=64) | MQAR recall | Improved |
| increase softmax regressor order from p=0 to p=1 (local linear regression) (ablation) | nonparametric softmax regression layer · switching autoregressive online regression | one-step test loss | Improved |
| add exponential weighting/forgetting to regression instead of unweighted RLS (ablation) | RLS layer · switching autoregressive online regression | one-step test loss | Improved |
| use linear attention instead of RLS when keys are correlated | parametric regression layer · switching autoregressive online regression | one-step test loss | Hurt |
| increase MQAR sequence length with large enough memory (ablation) | linear attention with orthonormal keys · MQAR (P=64, d_model=64) | MQAR recall | No clear effect |
Closest papers by meaning
- On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing (NeurIPS 2026 · similarity 0.88)
- Raven: High-Recall Sequence Modeling via Sparse Memory Routing (NeurIPS 2026 · similarity 0.88)
- RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.88)
- Zoology: Measuring and Improving Recall in Efficient Language Models (ICLR 2024 · similarity 0.88)
- Titans: Learning to Memorize at Test Time (NeurIPS 2025 · similarity 0.87)
- Overcoming Long Context Limitations of State Space Models via Context Dependent Sparse Attention (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
