Research map / Large language models / Retrieval and knowledge
MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MEMOIR, a lifelong model editing method that adds a zero-initialized residual memory layer to one FFN projection and sparsifies its inputs with sample-dependent TopHash masks, so each edit updates a distinct column subset and inference routes queries by mask overlap.
Problem
Lifelong model editing of LLMs must sequentially inject new facts while keeping reliability on edits, generalization to paraphrases, and locality on unrelated inputs, without catastrophic forgetting over thousands of edits.
Approach
Sparsify input activations of a residual memory layer with sample-dependent top-k masks plus a fixed random permutation (TopHash), so each edit writes to a distinct column subset; at inference, compare masks by overlap ratio to route rephrased queries to the matching edit and deactivate the memory for irrelevant inputs.
Key results
- On ZsRE with 1,000 edits on LLaMA-3-8B, MEMOIR reaches average 0.93 (Rel 0.94, Gen 0.85, Loc 1.00), outperforming the next best method by 0.14.
- On SelfCheckGPT with 600 edits on LLaMA-3, MEMOIR keeps locality at 1.00 and perplexity 61% lower than WISE.
- On LLaMA-3 with 15,000 edits, MEMOIR averages 0.88 while GRACE averages 0.76 and WISE 0.61.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MEMOIR instead of WISE | LLaMA-3-8B · ZsRE, T=1,000 | Avg.: 0.77 -> 0.93 | Helped |
| Use MEMOIR instead of WISE | Mistral-7B · ZsRE, T=1,000 | Avg.: 0.79 -> 0.94 | Helped |
| Use MEMOIR instead of WISE | LLaMA-3-8B · SelfCheckGPT, T=600 | Rel. (PPL): 3.20 -> 1.25 | Helped |
| Use MEMOIR instead of WISE | Mistral-7B · SelfCheckGPT, T=600 | Rel. (PPL): 5.21 -> 1.22 | Helped |
| Use MEMOIR instead of WISE | GPT-J-6B · Temporal, T=10 | Avg.: 0.78 -> 0.79 | Helped |
| Use MEMOIR instead of WISE | GPT-J-6B · Temporal, T=75 | Avg.: 0.78 -> 0.79 | Helped |
| Use MEMOIR instead of GRACE | LLaMA-3 · ZsRE, T=15,000 | Avg.: 0.76 -> 0.88 | Helped |
| Add conditional knowledge activation (ablation) | MEMOIR · ZsRE, T=1,000, LLaMA-3 | Loc.: 0.68 -> 1.00 | Helped |
| Add conditional knowledge activation (ablation) | MEMOIR · ZsRE, T=1,000, LLaMA-3 | Gen.: 0.77 -> 0.85 | Helped |
| Add activation centering using 100 irrelevant prompts (ablation) | MEMOIR · ZsRE, T=1,000, LLaMA-3 | Avg.: 0.92 -> 0.93 | Helped |
| Use TopHash instead of TopK selection (ablation) | MEMOIR · ZsRE, LLaMA-3 | Avg.: 0.90 -> 0.93 | Helped |
| Use TopHash instead of random index selection (ablation) | MEMOIR · ZsRE, LLaMA-3 | Avg.: 0.54 -> 0.93 | Helped |
Surprising findings
- Without the conditional knowledge activation, locality drops to 0.68 and average to 0.80, showing routing is essential for locality.
- Pure random index selection yields reliability of only 0.31 on ZsRE, indicating semantically grounded masks are necessary.
Limitations
- Modifies only a single linear layer, which may limit handling of long-horizon or broader knowledge changes.
- Experiments restricted to decoder-only transformers; multimodal and encoder-decoder models not tested.
- Relies on a threshold tau for routing between edited and pre-trained knowledge.
Experimental setup
Datasets: ZsRE SelfCheckGPT Temporal The Pile RippleEdits
Benchmarks: ZsRE SelfCheckGPT Temporal RippleEdits
Metrics: reliability generalization locality perplexity OOD generalization average score
Baselines: FT ROME MEMIT DEFER GRACE WISE AlphaEdit
Builds on: ROME MEMIT AlphaEdit WISE GRACE
Closest papers by meaning
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.94)
- Towards Lifelong Model Editing via Simulating Ideal Editor (ICML 2025 · similarity 0.91)
- ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA (AAAI 2025 · similarity 0.91)
- Reinforced Lifelong Editing for Language Models (ICML 2025 · similarity 0.91)
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.90)
- Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.