Research map / Large language models / Retrieval and knowledge
WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WISE, a dual parametric memory (frozen main FFN value matrix plus an edited side copy with routing and knowledge-sharding merges) for lifelong model editing of LLMs, reporting gains over ROME, MEMIT, FT-EWC, MEND, DEFER and GRACE on ZsRE, SelfCheckGPT hallucination, and Temporal OOD tasks.
Problem
Lifelong model editing of LLMs requires simultaneously high reliability, generalization, and locality; existing long-term parametric editors cause interference with pretrained knowledge, while retrieval-based working-memory editors fail to generalize.
Approach
Keep the original FFN value matrix as main memory and edit only a copy (side memory) in random gradient-masked subspaces, merged via TIES; a learned activation-based router decides which memory to use per query, so edits are isolated from pretrained knowledge while remaining parametric and generalizable.
Key results
- On ZsRE with T=1000 edits, WISE averages 0.83 on LLaMA-2-7B and 0.79 on Mistral-7B, with locality 1.00 in both.
- On SelfCheckGPT hallucination with T=600, WISE attains perplexity 3.12 (LLaMA-2-7B) and 5.21 (Mistral-7B) with locality above 0.93.
- On Temporal OOD (GPT-J-6B, T=75), WISE reaches OOD generalization 0.37 and average 0.78, versus 0.75 for GRACE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WISE instead of MEMIT-MASS for continual editing | LLaMA-2-7B · ZsRE, T=1000 | Avg.: 0.65 -> 0.83 | Helped |
| Use WISE instead of GRACE for continual editing | LLaMA-2-7B · ZsRE, T=1000 | Avg.: 0.67 -> 0.83 | Helped |
| Use WISE instead of MEMIT-MASS for continual editing | Mistral-7B · ZsRE, T=1000 | Avg.: 0.68 -> 0.79 | Helped |
| Use WISE instead of GRACE for continual editing | Mistral-7B · ZsRE, T=1000 | Avg.: 0.67 -> 0.79 | Helped |
| Use WISE instead of MEMIT-MASS for hallucination editing | LLaMA-2-7B · SelfCheckGPT, T=600 | PPL: 13.47 -> 3.12 | Helped |
| Use WISE instead of GRACE for hallucination editing | Mistral-7B · SelfCheckGPT, T=600 | PPL: 9.57 -> 5.21 | Helped |
| Use WISE instead of GRACE for out-of-distribution editing | GPT-J-6B · Temporal, T=75 | OOD Gen.: 0.28 -> 0.37 | Helped |
| Retrieve among multiple side memories rather than repeatedly merge into one (ablation) | WISE · ZsRE, T=3000 | Avg.: 0.71 -> 0.73 | Helped |
| Add the routing activation loss L_a (ablation) | WISE · ZsRE, T=1000 | Loc.: 0.72 -> 1.00 | Helped |
| Add the routing activation loss L_a (ablation) | WISE · ZsRE, T=1000 | Avg.: 0.78 -> 0.83 | Helped |
| Edit a mid-to-late layer (layer 26) instead of early or final layers (ablation) | WISE · ZsRE | Locality success rate: 0.096 -> 100% | Helped |
| Set mask ratio rho to 0.2 when k=2 (ablation) | WISE · ZsRE | Avg. performance of Rel., Gen., and Loc. | Helped |
Surprising findings
- GRACE's retrieval-based codebook achieves near-perfect reliability and locality but collapses on generalization (about 0.15 at T=100 on ZsRE), with token representations poorly suited to paraphrase matching in decoder-only LLMs.
- Editing very early or final layers (0, 1, 31) yields very low locality (0.096), contrary to the intuition that they would be natural edit sites.
Limitations
- WISE-Retrieve's side-memory retrieval falls short of the oracle routing, indicating room for improvement.
- WISE-Retrieve inference time increases as the number of edits grows.
- Some very long hallucination edits could not fit in GPU memory, so the dataset was filtered from 1392 to 906 instances.
Experimental setup
Datasets: ZsRE SelfCheckGPT (GRACE hallucination set) Temporal NQ RedPajama Pile
Benchmarks: ZsRE SelfCheckGPT Temporal
Metrics: reliability generalization locality perplexity OOD generalization success rate average score
Baselines: FT-L FT-EWC MEND ROME MEMIT MEMIT-MASS DEFER GRACE
Builds on: ROME MEMIT MEND GRACE TIES-Merging Elastic Weight Consolidation
Closest papers by meaning
- MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs (NeurIPS 2025 · similarity 0.94)
- Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models (AAAI 2025 · similarity 0.91)
- Editing Memories Through Few Targeted Neurons (AAAI 2025 · similarity 0.91)
- History Matters: Temporal Knowledge Editing in Large Language Model (AAAI 2024 · similarity 0.91)
- ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA (AAAI 2025 · similarity 0.91)
- Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.