Research map / Architectures and efficiency / Attention and transformers
Can Active Memory Replace Attention?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition each output symbol on the previous output (ablation) | Neural GPU · WMT'14 English-French newstest'14 | log per-word perplexity: 3.5 -> 2.5 | Helped |
| Add a recurrent active-memory decoder with access to prior output embeddings (ablation) | Markovian Neural GPU · WMT'14 English-French newstest'14 | log per-word perplexity: 2.5 -> 1.19 | Helped |
| Add a recurrent active-memory decoder with access to prior output embeddings (ablation) | Markovian Neural GPU · WMT'14 English-French newstest'14 | BLEU: < 5 -> 29.6 | Helped |
| Use an output-length estimator based on input sentence length (ablation) | Extended Neural GPU · WMT'14 English-French translation | BLEU: -0.3 BLEU | Hurt |
| Use active memory rather than GRU attention | Extended Neural GPU versus GRU+Attention · WMT'14 English-French newstest'14 | BLEU: 26.4 -> 29.6 | Helped |
| Use active memory rather than GRU attention | Extended Neural GPU versus GRU+Attention · WMT'14 English-French newstest'14 | per-word perplexity: 3.4 -> 3.3 | No clear effect |
| Use active memory rather than GRU attention | Extended Neural GPU versus GRU+Attention · WMT'14 English-French, by source sentence length | BLEU | Helped |
| Use Extended Neural GPU on constituency parsing | Extended Neural GPU · WSJ 23 | F1: 88.3 -> 85.1 | Hurt |
Closest papers by meaning
- Proteus: Incremental Memory Activation for Long-Context Sequence Modeling (NeurIPS 2026 · similarity 0.85)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.84)
- Titans: Learning to Memorize at Test Time (NeurIPS 2025 · similarity 0.84)
- Selective Attention Improves Transformer (ICLR 2025 · similarity 0.84)
- Zoology: Measuring and Improving Recall in Efficient Language Models (ICLR 2024 · similarity 0.84)
- When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.