Research map / Large language models / Retrieval and knowledge
AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AlphaEdit, which projects locate-then-edit weight perturbations onto the null space of preserved-knowledge keys so that sequential model edits do not disturb preserved knowledge, adding one projection line to existing editors.
Problem
Locate-then-edit model editing trades off knowledge-update error against knowledge-preservation error; in sequential editing this overfitting causes forgetting and model collapse.
Approach
Remove the preserved-knowledge error term from the editing objective and instead project the perturbation onto the null space of the preserved keys (via the projection matrix from SVD of K0 K0^T), guaranteeing (W+ΔP)K0 = W K0 = V0, plus a regularizer and a term protecting previously edited keys.
Key results
- On LLaMA3 with 2,000 sequential edits, AlphaEdit reaches 98.90% Efficacy on Counterfact versus 65.65% for MEMIT.
- AlphaEdit averages a 36.7% improvement in editing capability when the projection is added to baseline editors.
- Baseline-edited LLaMA3 general capability metrics degrade toward zero after 2,000 edits, while AlphaEdit maintains original performance up to 3,000 edits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add null-space projection to the editing update | locating-then-editing methods · sequential editing; Counterfact and ZsRE | Efficacy and generalization: 12.54%; 16.78% | Helped |
| Add null-space projection to the editing update | locating-then-editing methods · sequential editing; Counterfact and ZsRE | Editing capability: 28.24% | Helped |
| Add null-space projection to the editing update | PRUNE and RECT · general capability evaluations | General capability: 42.65% | Helped |
| Add null-space projection to the editing update | AlphaEdit-edited LLMs · GLUE general capability tasks after editing 3,000 samples | general capability | No clear effect |
| Add null-space projection to the editing update | AlphaEdit · Counterfact; GPT2-XL (1.5B) | Fluency: 18.33% | Helped |
| Use null-space projection rather than the unprojected MEMIT update | MEMIT · sequential editing; multiple benchmarks and models | editing performance: 36.7% | Helped |
Surprising findings
- Adding only the projection line to MEMIT, PRUNE and RECT improves their general capability by an average of 42.65%.
- Baseline editors can reverse the distribution trend of hidden representations after editing (e.g., RECT on LLaMA3).
Limitations
- Applicability to multimodal LLMs and large reasoning models is unexplored.
- Relies on estimating K0 from 100,000 Wikipedia triplets, which only approximates the model's full preserved knowledge.
Experimental setup
Datasets: Counterfact ZsRE GLUE (SST, MRPC, CoLA, RTE, NLI) MMLU KnowEdit MQuAKE LongformEvaluation
Benchmarks: Counterfact ZsRE GLUE MMLU
Metrics: Efficacy Generalization Specificity Fluency Consistency F1 accuracy
Baselines: FT MEND InstructEdit ROME MEMIT PRUNE RECT
Builds on: MEMIT ROME Adam-NSCL PRUNE RECT
Closest papers by meaning
- UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models (NeurIPS 2025 · similarity 0.93)
- AnyEdit: Edit Any Knowledge Encoded in Language Models (ICML 2025 · similarity 0.92)
- Should We Really Edit Language Models? On the Evaluation of Edited Language Models (NeurIPS 2024 · similarity 0.91)
- Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.91)
- Towards Lifelong Model Editing via Simulating Ideal Editor (ICML 2025 · similarity 0.91)
- Unveiling the Pitfalls of Knowledge Editing for Large Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.