Research map / Large language models / Retrieval and knowledge
MEMORYLLM: Towards Self-Updatable Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MemoryLLM, a Llama2-7B augmented with a ~1B-parameter fixed-size memory pool of per-layer memory tokens that is self-updated by the transformer itself with random dropping for exponential forgetting.
Problem
LLMs remain static after deployment, making it hard to inject new knowledge efficiently without catastrophic forgetting or unbounded growth of retrieval stores or contexts.
Approach
Embed a fixed-size memory pool of hidden memory tokens in every transformer layer; new text is absorbed by running the transformer over the last K memory tokens plus the new context, then replacing K randomly dropped memory tokens with the new ones, giving exponential forgetting without backprop-based editing.
Key results
- On CounterFactual, MemoryLLM-7B (w/ EF) achieves a Score of 75.3 vs 69.2 for ROME and 70.7 for IKE.
- On zsRE, MemoryLLM-7B (w/ EF) reaches Score 79.2 vs 69.8 for FT-L and 69.3 for ROME, with 99.8 efficacy.
- No accuracy decrease observed over 650k self-update steps on SQuAD and NaturalQA knowledge-retention integrity checks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Update the memory with an editing fact | MemoryLLM-7B · ZsRE | Score: 55.6 -> 79.2 | Helped |
| Update the memory with an editing fact | MemoryLLM-7B · CounterFactual | Score: 20.7 -> 75.3 | Helped |
| Use BM25 to retrieve 4k tokens before MemoryLLM processing | MemoryLLM · Long-context QA benchmarks | Score: 20.64 -> 15.60; 19.57 -> 20.30; 29.56 -> 33.08; 34.03 -> 32.27; 27.22 -> 24.17; 13.47 -> 15.36 | Mixed |
| Increase memory size N while holding K fixed (ablation) | MemoryLLM · NaturalQA and SQuAD knowledge-retention experiments | knowledge-retention ratio | Helped |
| Decrease K while holding N fixed (ablation) | MemoryLLM · NaturalQA and SQuAD knowledge-retention experiments | knowledge-retention ratio | Helped |
| Use K=128 memory tokens per update (ablation) | MemoryLLM · NaturalQA | step 1 accuracy: 0.34 | Hurt |
| Use K=128 memory tokens per update (ablation) | MemoryLLM · SQuAD | step 1 accuracy: 0.25 | Hurt |
| Augment only one transformer layer with memory tokens (ablation) | MemoryLLM · Context-based evaluation | accuracy | No clear effect |
| Augment only the last half of layers with memory tokens (ablation) | MemoryLLM · NaturalQA | accuracy: 0.46 -> 0.39 | Hurt |
| Augment only the last half of layers with memory tokens (ablation) | MemoryLLM · SQuAD | accuracy: 0.39 -> 0.22 | Hurt |
Surprising findings
- Unedited MemoryLLM-7B scores lower than Llama2-7B on zsRE (51.2 vs 55.6), indicating the memory architecture alone slightly degrades base performance.
- MemoryLLM underperforms on Qasper, which the authors attribute to training only on C4 without arxiv data.
Limitations
- Performance on scientific long-context QA (Qasper) is weaker, attributed to C4-only training data.
- Knowledge retention falls short of the theoretical exponential decay upper bound in knowledge-retention experiments.
- Long-context and editing comparisons are against non-instruction-tuned baselines; MEND was omitted due to NaN issues on Llama2.
Experimental setup
Datasets: C4 RedPajama SQuAD NaturalQA zsRE CounterFactual
Benchmarks: zsRE CounterFactual LongBench SQuAD NaturalQA
Metrics: harmonic mean score efficacy generalization specificity F1 accuracy
Baselines: FT FT-L IKE ROME Llama2-7B LongLLaMA-3B LongLoRA
Builds on: Llama2 ROME MemoryBank Memory Transformer
Closest papers by meaning
- M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.92)
- Self-Updatable Large Language Models by Integrating Context into Model Parameters (ICLR 2025 · similarity 0.91)
- Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.89)
- Peripheral Memory for LLMs: Integration of Sequential Memory Banks with Adaptive Querying (ICML 2025 · similarity 0.89)
- From Static to Dynamic: Knowledge Metabolism for Large Language Models (AAAI 2024 · similarity 0.88)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.