Research map / Architectures and efficiency / RNNs and normalization
Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Memoria, a human-inspired external memory system (working, short-term, and graph-linked long-term memory with Hebbian edge weights and lifespan-based forgetting) that is integrated into Transformers and BERT/RoBERTa to address "fateful forgetting" in long sequences.
Problem
Neural networks with external memory tend to keep only recent information and forget important old information over time (fateful forgetting), with fixed-capacity memories and weak cue-based retrieval of long-term memories.
Approach
Memoria organizes engrams into working, short-term, and long-term memory; long-term engrams are linked in a directed weighted graph updated by Hebbian co-retrieval counts, retrieved via DFS from short-term cues, and given lifespans proportional to their attention-based contribution so that useful memories persist while unused ones decay.
Key results
- Memoria Transformer reaches 23.471 perplexity on WikiText-103 versus 24.543 for Transformer-XL and 26.755 for the plain Transformer (segment length 150).
- On Hyperpartisan test, Memoria RoBERTa achieves 96.51 F1 versus 95.24 for RoBERTa, with one-tailed t-test p-values of 0.045 vs Longformer and 0.005 vs BigBird.
- Engram analysis shows primacy and recency effects and temporal contiguity effects in remaining long-term engrams on WikiText-103.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Memoria Transformer instead of Transformer-XL | language modeling · WikiText-103 | PPL: 24.543 -> 23.471 | Helped |
| Use Memoria Transformer instead of Compressive Transformer | language modeling · PG-19 | PPL: 29.603 -> 29.149 | Helped |
| Use Memoria Transformer instead of Compressive Transformer | language modeling · Enwik8 | BPC: 1.16 -> 1.16 | No clear effect |
| Add Memoria to RoBERTa | RoBERTa · Hyperpartisan validation | F1: 82.96 ± 0.02 -> 86.39 ± 0.01 | Helped |
| Add Memoria to RoBERTa | RoBERTa · Hyperpartisan test | accuracy: 95.38 ± 0.02 -> 96.62 ± 0.02 | Helped |
| Use Memoria Transformer instead of ∞-former | sorting model · Sorting, 32 segments, segment length 1024 | accuracy: 39.71 -> 63.42 | Helped |
| Add working memory (ablation) | Transformer sorting model · Sorting, 48K | accuracy: 19.04 -> 34.32 | Helped |
| Add short-term memory to working memory (ablation) | Transformer sorting model · Sorting, 48K | accuracy: 34.32 -> 54.87 | Helped |
| Add long-term memory to working and short-term memory (ablation) | Transformer sorting model · Sorting, 48K | accuracy: 54.87 -> 63.26 | Helped |
| Add long-term memory (ablation) | Memoria Transformer · Sorting, 48K | accuracy: 54.87 -> 63.26 | Helped |
| Replace Hebbian connection updates with random wiring (ablation) | Memoria Transformer · Sorting, 48K | accuracy: 63.26 -> 56.13 | Hurt |
| Search all long-term memory without internal graph weights (ablation) | Memoria Transformer · Sorting, 48K | accuracy: 63.26 -> 65.84 | Helped |
Surprising findings
- Random wiring of long-term memory edges reduces the long-term memory gain from +8.39 to +1.26 on 48K sorting, while full LTM search gives +10.97 but at much higher cost.
- On the 4K sorting setting, long-term memory slightly hurts accuracy (-0.39) and the 8K setting is also negative (-2.12).
Limitations
- Direct comparison with human memory is limited; effects are interpreted qualitatively.
- Classification experiments rely on pre-trained models due to pre-training cost, so comparisons are confounded by pre-training differences.
- Engram design is restricted to embedding vectors in experiments; text-based engrams are only described theoretically.
Experimental setup
Datasets: WikiText-103 PG-19 enwik8 Hyperpartisan synthetic sorting data (generated with infinite-former script)
Benchmarks: WikiText-103 PG-19 enwik8 Hyperpartisan sorting task (1K-32K sequences)
Metrics: perplexity bits-per-character accuracy macro F1 accuracy (sorting)
Baselines: Transformer Transformer-XL Compressive Transformer infinity-former BERT RoBERTa Longformer BigBird
Builds on: Transformer-XL Compressive Transformer Multi-Store model SAM Hebbian learning
Closest papers by meaning
- Predictive Attractor Models (NeurIPS 2024 · similarity 0.86)
- Memorization Sinks: Isolating Memorization during LLM Training (ICML 2025 · similarity 0.85)
- AUTOMEM: Automated Learning of Memory as a Cognitive Skill (NeurIPS 2026 · similarity 0.84)
- Think Before You Act: Decision Transformers with Working Memory (ICML 2024 · similarity 0.84)
- Simulating Human Memory with Language Models (NeurIPS 2026 · similarity 0.84)
- Factual recall in linear associative memories: sharp asymptotics and mechanistic insights (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.