Research map / Large language models / Retrieval and knowledge
Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NMKE, a neuron-level lifelong knowledge editing method that uses attribution-based knowledge-general and knowledge-specific neuron selection with an entropy-guided dynamic sparse mask on top of AlphaEdit's null-space pipeline.
Problem
Lifelong knowledge editing in LLMs accumulates errors over thousands of sequential edits, causing declining edit success, generalization, and loss of general capabilities when layer-level parameter updates disrupt unrelated neurons.
Approach
Attributes FFN neuron importance via log-probability gain under perturbation, classifies neurons as knowledge-general (consistently positive across prompts) or knowledge-specific (large max attribution), and selects per-batch neuron subsets with an entropy-derived ratio to mask the AlphaEdit update matrix.
Key results
- On ZsRE with LLaMA3-8B-Instruct at T=2000, NMKE reaches Rel. 0.94, Gen. 0.85, Loc. 0.71 versus AlphaEdit's 0.32/0.28/0.06 at T=1500 and 0.62/0.54/0.14 at T=2000 in the reported table.
- At T=5000 on ZsRE, NMKE retains Rel. 0.86 and Gen. 0.74, while AlphaEdit drops to 0.02 and 0.02.
- In batched editing (batch size 4) at T=2000, NMKE gets Rel. 0.90 and Gen. 0.80 versus AlphaEdit's 0.68 and 0.58.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace AlphaEdit with NMKE | LLaMA3-8B-Instruct · ZsRE, T=2000 | Rel.: 0.32 -> 0.94 | Helped |
| Replace AlphaEdit with NMKE | LLaMA3-8B-Instruct · CounterFact, T=2000 | Rel.: 0.22 -> 0.98 | Helped |
| Replace AlphaEdit with NMKE | LLaMA3-8B-Instruct · ZsRE, T=5000 | Rel.: 0.02 -> 0.86 | Helped |
| Use NMKE instead of AlphaEdit | LLaMA3-8B-Instruct · ZsRE, T=2000 | Loc.: 0.06 -> 0.71 | Helped |
| Use NMKE instead of AlphaEdit | LLaMA3-8B-Instruct · CounterFact, T=2000 | Gen.: 0.13 -> 0.67 | Helped |
| Use a soft mask instead of a binary sparse mask (ablation) | NMKE · ZsRE, T=2000 | Loc.: 0.74 -> 0.19 | Hurt |
| Use only overlapping neurons (ablation) | NMKE · Sequential editing, T=2000 | Rel.: 0.94 -> 0.75 | Hurt |
| Use only non-overlapping neurons (ablation) | NMKE · Sequential editing, T=2000 | Rel.: 0.94 -> 0.85 | Hurt |
| Mask top-10 knowledge-general neurons (ablation) | LLaMA2-7B · MMLU high school chemistry | Accuracy (%): 36.94 -> 1.48 | Hurt |
| Mask top-10 task-relevant neurons (ablation) | LLaMA2-7B · MMLU high school chemistry | Accuracy (%): 36.94 -> 22.66 | Hurt |
| Use NMKE instead of AlphaEdit | GPT2-XL · ZsRE, T=2000 | Rel.: 0.89 -> 0.90 | Helped |
| Use NMKE instead of AlphaEdit | Qwen2.5-7B · ZsRE, T=2000 | Rel.: 0.92 -> 0.96 | Helped |
Surprising findings
- Masking only the top-10 knowledge-general neurons drops MMLU college biology accuracy from 37.5% to 4.17%, while masking task-specific neurons causes only minor drops.
- Soft masking (scaling updates by attribution) matches NMKE on reliability and generalization but severely degrades locality (0.19 at T=2000).
Limitations
- Neuron labeling is binary (general vs specific) based on activation patterns, ignoring the continuous nature of neuron responses and cross-layer interactions.
- Edits do not model dependency closure across related facts (e.g., consistency of dependent facts after an update).
- On CounterFact, NMKE's generalization is lower than AlphaEdit's at early steps (e.g., 0.51 vs 0.60 at T=10).
Experimental setup
Datasets: ZsRE CounterFact MMLU GSM8K CommonsenseQA BBH HumanEval
Benchmarks: ZsRE CounterFact MMLU GSM8K CommonsenseQA BBH-Zeroshot HumanEval
Metrics: Rel. (edit success) Gen. (generalization) Loc. (locality) accuracy
Baselines: FT KN ROME PMET MEMIT WISE AlphaEdit
Builds on: AlphaEdit ROME MEMIT FiNE EasyEdit
Closest papers by meaning
- Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs (ICLR 2025 · similarity 0.91)
- Editing Memories Through Few Targeted Neurons (AAAI 2025 · similarity 0.90)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.89)
- Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
- MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs (NeurIPS 2025 · similarity 0.88)
- Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.