Research map / Large language models / Language, safety and interpretability
Rethinking Residual Distribution in Locate-then-Edit Model Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that residual distribution in locate-then-edit methods causes weight-shift errors that grow with distribution distance, batch size and edit count, and proposes BLUE, which updates only the first and last critical layers with directly computed residuals.
Problem
Locate-then-edit model editing methods distribute residuals evenly across critical layers, which introduces weight-shift errors that degrade editing precision in sequential batch editing of LLMs.
Approach
Replace even residual distribution with direct residual computation at only the first and last critical layers (BLUE), which theory and experiments suggest reduces weight-shift error and improves editing, retention and efficiency when plugged into existing locate-then-edit methods.
Key results
- BLUE applied to locate-then-edit methods yields an average improvement of 35.59% over original methods; 86 of 96 sequential batch results are enhanced.
- On Llama3 (8B), PRUNE_BLUE improves CounterFact efficacy from 68.25 to 96.73 and generalization from 64.75 to 89.68 in sequential batch editing.
- BLUE-enhanced MEMIT and PRUNE retain general capabilities across 3,000 sequential edits on Llama3 where the originals nearly collapse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply BLUE, updating the first and last critical layers with directly computed residuals | AlphaEdit on Llama3-8B · CounterFact sequential batch | Efficacy: 98.90 -> 99.93 | Helped |
| Apply BLUE, updating the first and last critical layers with directly computed residuals | AlphaEdit on Llama3-8B · ZsRE sequential batch | Efficacy: 94.47 -> 95.77 | Helped |
| Apply BLUE, updating the first and last critical layers with directly computed residuals | AlphaEdit on GPT-J (6B) · CounterFact sequential batch | Efficacy: 99.75 -> 99.77 | No clear effect |
| Apply BLUE, updating the first and last critical layers with directly computed residuals | AlphaEdit on GPT-J (6B) · ZsRE sequential batch | Efficacy: 99.79 -> 99.63 | Hurt |
| Apply BLUE, updating the first and last critical layers with directly computed residuals | AlphaEdit on GPT2-XL · CounterFact sequential batch | Efficacy: 99.50 -> 99.40 | Hurt |
| Apply BLUE, updating the first and last critical layers with directly computed residuals | AlphaEdit on GPT2-XL · ZsRE sequential batch | Efficacy: 94.81 -> 96.88 | Helped |
| Apply BLUE to MEMIT | MEMIT on Llama3-8B · UnKEBench | BLEU (original questions): 77.65 -> 82.34 | Helped |
| Select the first and last critical layers (layers 4 and 8) rather than other layer pairs (ablation) | BLUE-enhanced editing methods on Llama3-8B · CounterFact sequential editing | editing performance | Helped |
| Use directly computed residuals instead of distributed residuals | Locate-then-edit methods · Single-layer editing | Efficacy: over 3× | Helped |
| Use directly computed residuals instead of distributed residuals | Locate-then-edit methods · Single-layer editing | Generalization: exceeds 2× | Helped |
| Increase residual distribution distance | Locate-then-edit methods · Model editing | editing contribution score | Hurt |
| Increase editing batch size | Locate-then-edit methods using residual distribution · Model editing | weight shift error | Hurt |
Surprising findings
- Even distributing residuals across just one layer causes a significant drop in contribution score, with the first critical layer below 0.1 in three LLMs.
- BLUE improves AlphaEdit only marginally on GPT-J (0.03% average), because AlphaEdit's baseline is already strong.
Limitations
- Gains are smaller for large-scale batch editing (10,000 edits), and BLUE-enhanced methods show lower specificity in some settings.
- Theoretical bound is an upper bound and does not establish the error itself; the lower bound is vacuous in practice, per the authors' remark.
- Less effective for PMET (square-root residual distribution), with limited gains in several settings.
Experimental setup
Datasets: CounterFact ZsRE GLUE tasks (SST, MRPC, MMLU, RTE, CoLA, NLI) UnKEBench
Benchmarks: CounterFact ZsRE UnKEBench GLUE
Metrics: efficacy generalization specificity fluency consistency F1 BLEU ROUGE BERTScore peak memory
Baselines: MEMIT PRUNE RECT AlphaEdit PMET
Builds on: MEMIT AlphaEdit PRUNE RECT
Closest papers by meaning
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.91)
- Towards Lifelong Model Editing via Simulating Ideal Editor (ICML 2025 · similarity 0.90)
- Should We Really Edit Language Models? On the Evaluation of Edited Language Models (NeurIPS 2024 · similarity 0.89)
- MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs (NeurIPS 2025 · similarity 0.89)
- Norm Anchors Make Model Edits Last (NeurIPS 2026 · similarity 0.89)
- Reasons and Solutions for the Decline in Model Performance after Editing (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.