Research map / Large language models / Retrieval and knowledge
Revealing and Mitigating Over-Attention in Knowledge Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies Attention Drift as the cause of Specificity Failure in locate-then-edit knowledge editing and proposes SADR, a regularizer that restrains attention-distribution changes in over-attending heads during editing.
Problem
Knowledge editing of LLMs can cause Specificity Failure: when the edited subject appears in the context, unrelated existing knowledge and capabilities are corrupted.
Approach
Attributes Specificity Failure to Attention Drift, where the last token's attention to the edited subject becomes excessive in some heads; SADR adds a KL-based regularizer restricting attention-distribution changes only in heads that over-focus on the edited subject during the value optimization.
Key results
- SADR applied to ROME on GPT-J raises Relation Score from 11.94 to 27.75 and Distract Neighborhood Score from 30.42 to 49.35, with ES dropping only from 99.88 to 99.76.
- Across five models (TinyLlama-1.1B to GPT-NeoX-20B) and three locate-then-edit editors, SADR yields improvements of up to 130.9% and 295.8% on the two main specificity tasks, per the abstract.
- With MEMIT on GPT-J, SADR raises Avg. S from 51.07 to 60.23 while keeping ES at 99.52 vs 99.64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SADR to ROME | ROME on GPT-J-6B · CounterFact + WikiData_counterfact | RS: 11.94 -> 27.75(2.3) | Helped |
| Add SADR to MEMIT | MEMIT on GPT-J-6B · CounterFact + WikiData_counterfact | RS: 25.78 -> 34.68(2.5) | Helped |
| Add SADR to PMET | PMET on GPT-J-6B · CounterFact + WikiData_counterfact | RS: 27.68 -> 33.33(2.5) | Helped |
| Add SADR to ROME | ROME on Llama3-8B · CounterFact + WikiData_counterfact | RS: 29.38 -> 47.67(2.6) | Helped |
| Add SADR to MEMIT | MEMIT on Llama3-8B · CounterFact + WikiData_counterfact | RS: 17.44 -> 35.38(2.5) | Helped |
| Add SADR to PMET | PMET on Llama3-8B · CounterFact + WikiData_counterfact | RS: 19.84 -> 34.68(2.5) | Helped |
| Add SADR to ROME | ROME on GPT-NeoX-20B · CounterFact + WikiData_counterfact | RS: 15.11 -> 34.89(2.5) | Helped |
| Add SADR to MEMIT | MEMIT on GPT-NeoX-20B · CounterFact + WikiData_counterfact | RS: 35.24 -> 45.48(2.6) | Helped |
| Add SADR to PMET | PMET on GPT-NeoX-20B · CounterFact + WikiData_counterfact | RS: 12.29 -> 24.86(2.3) | Helped |
| Restrict only heads with excessive drift rather than all heads (ablation) | SADR with ROME on GPT-J-6B · Edit Success and Specificity | average score | Helped |
| Vary SADR restraint weight gamma instead of adjusting original ROME hyperparameters (ablation) | SADR with ROME on GPT-J-6B · Edit Success–Specificity trade-off | trade-off | Helped |
| Disable attention-weight optimization during ROME target-vector optimization (ablation) | ROME-AWD on easy-to-edit knowledge · Distract Neighborhood | Specificity: 16.0 -> 40.3 | Helped |
Surprising findings
- Disabling attention-weight optimization in ROME (ROME-AWD) improves specificity on easy-to-edit facts, suggesting attention drift stems from editing hard-code knowledge into the forward pass via the subject's hidden states.
- Existing editing methods under standard evaluation (ES, PS, NS) miss specificity failures; ROME drops Relation Score from 79.73 to 11.94 on GPT-J while ES and PS stay near 100.
Limitations
- Relation task still shows room for improvement with SADR.
- Study limited to single-edit settings; batch and sequential editing not covered.
- Only transformer-based models are studied.
Experimental setup
Datasets: CounterFact WikiData_counterfact PIQA ME-PPL
Benchmarks: CounterFact WikiData_counterfact PIQA ME-PPL
Metrics: Efficacy Score Efficacy Magnitude Paraphrase Score Paraphrase Magnitude Neighborhood Score Neighborhood Magnitude Relation Score Relation Magnitude Distract Neighborhood Score Distract Neighborhood Magnitude
Baselines: ROME MEMIT PMET
Builds on: ROME MEMIT PMET causal tracing EasyEdit
Closest papers by meaning
- Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.91)
- Unveiling the Pitfalls of Knowledge Editing for Large Language Models (ICLR 2024 · similarity 0.90)
- Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models (AAAI 2025 · similarity 0.89)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.89)
- In-Context Editing: Learning Knowledge from Self-Induced Distributions (ICLR 2025 · similarity 0.89)
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.