Research map / Architectures and efficiency / Pruning and efficient tuning
Wasserstein Distance Constraint and Parameter Sparsification for Batched and Iterative Knowledge Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Wasserstein-distance loss and residual-update sparsification (pruning plus dropout) to stabilize batched iterative knowledge editing of parameter-updating methods such as MEMIT and PMET.
Problem
Parameter-updating knowledge editors (MEMIT, PMET) degrade and collapse when edits are applied in batched, iterative fashion because accumulated weight updates shift the parameter distribution.
Approach
Add a Sinkhorn-approximated Wasserstein-2 distance between initial and edited parameters as a dynamically scaled loss term when computing residual updates, and sparsify residuals via magnitude pruning and dropout so fewer parameters change per iteration.
Key results
- On zsRE (batch 50, 100 iterations), MEMIT average Score 39.66 rises to 54.67 with WD+SP; PMET 28.87 rises to 45.98 with WD+SP.
- On CounterFact (batch 50, 50 iterations), MEMIT average Score 84.17 vs 84.40 with WD+SP; PMET 81.56 vs 83.44 with WD+SP.
- On CounterFact (batch 100, 100 iterations), MEMIT average Score 75.02 rises to 79.03 with WD+SP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Wasserstein distance constraint and parameter update sparsification to MEMIT (ablation) | MEMIT · zsRE, batch size 50, 100 iterations | overall editing score: 39.66 -> 54.67 | Helped |
| Add Wasserstein distance constraint and parameter update sparsification to PMET (ablation) | PMET · zsRE, batch size 50, 100 iterations | overall editing score: 28.87 -> 45.98 | Helped |
| Add Wasserstein distance constraint to MEMIT (ablation) | MEMIT · zsRE, batch size 50, 100 iterations | average efficacy: 76.41 -> 99.94 | Helped |
| Add Wasserstein distance constraint and parameter update sparsification to MEMIT (ablation) | MEMIT · zsRE, batch size 50, 100 iterations | average generalization: 73.69 -> 96.85 | Helped |
| Add Wasserstein distance constraint and parameter update sparsification to MEMIT (ablation) | MEMIT · zsRE, batch size 50, 100 iterations | average locality: 21.33 -> 29.29 | Helped |
| Add parameter update sparsification to PMET (ablation) | PMET · zsRE, batch size 50, 100 iterations | average efficacy: 52.91 -> 74.08 | Helped |
| Add Wasserstein distance constraint and parameter update sparsification to PMET (ablation) | PMET · zsRE, batch size 50, 100 iterations | average efficacy: 52.91 -> 91.84 | Helped |
| Add Wasserstein distance constraint to MEMIT (ablation) | MEMIT · CounterFact, batch size 50, 50 iterations | overall editing score: 84.17 -> 84.12 | No clear effect |
| Add parameter update sparsification to MEMIT (ablation) | MEMIT · CounterFact, batch size 50, 50 iterations | overall editing score: 84.17 -> 83.83 | Hurt |
| Add Wasserstein distance constraint and parameter update sparsification to MEMIT (ablation) | MEMIT · CounterFact, batch size 50, 50 iterations | overall editing score: 84.17 -> 84.40 | No clear effect |
| Add Wasserstein distance constraint and parameter update sparsification to PMET (ablation) | PMET · CounterFact, batch size 50, 50 iterations | overall editing score: 81.56 -> 83.44 | Helped |
| Add Wasserstein distance constraint and parameter update sparsification to PMET (ablation) | PMET · CounterFact, batch size 50, 50 iterations | average locality: 61.84 -> 66.65 | Helped |
Surprising findings
- Discarding roughly 50% of update parameters does not degrade and may even improve editing performance.
- Baseline MEMIT and PMET collapse to near-zero editing score after roughly 70-90 batched iterations on zsRE.
Limitations
- Evaluated only on GPT-J 6B with two datasets and two parameter-updating baselines.
- Trade-off between Generalization and Locality remains; PMET combined method sacrifices some Generalization on CounterFact.
- Gains are not uniform; independent methods sometimes beat the combined approach in early iterations.
Experimental setup
Datasets: zsRE CounterFact
Benchmarks: zsRE CounterFact
Metrics: Efficacy Generalization Locality harmonic-mean editing Score
Baselines: MEMIT PMET
Builds on: MEMIT PMET ROME
Closest papers by meaning
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.89)
- Should We Really Edit Language Models? On the Evaluation of Edited Language Models (NeurIPS 2024 · similarity 0.89)
- UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models (NeurIPS 2025 · similarity 0.88)
- Rethinking Residual Distribution in Locate-then-Edit Model Editing (NeurIPS 2025 · similarity 0.88)
- Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.88)
- Reasons and Solutions for the Decline in Model Performance after Editing (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.