Research map / Large language models / Retrieval and knowledge
SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SWEA⊕OS, a detachable model-editing framework that matches subject token IDs and adds optimized, KED-suppressed editing embeddings to subject word embeddings, evaluated on CounterFact, zsRE and RippleEdits with GPT-J and Llama-2.
Problem
Local model editing methods that update FFN weights via least squares are computationally expensive, rely on unreliable vector-level matching to recognize edited knowledge, and disrupt the original parameter organization.
Approach
Instead of editing weights, it caches learnable editing embeddings keyed by subject token IDs (token-level matching) and adds them to subject word embeddings at input. Embeddings are obtained by optimizing learnable vectors, then suppressing subject-specific knowledge embedding dimensions (KEDs) identified via knowledge attribution.
Key results
- On CounterFact with 10,000 edits, SWEA⊕OS reaches Score 91.2 on GPT-J and 89.6 on Llama-2, versus 86.2 and 83.2 for the best baseline PMET.
- On zsRE (10,000 edits), SWEA⊕OS achieves Score 51.0 on GPT-J and 25.5 on Llama-2, the best among compared editors.
- SWEA⊕OS cuts 10,000-edit execution time by 47.8% on GPT-J and 17.6% on Llama-2 relative to MEMIT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SWEA⊕OS instead of PMET | GPT-J · CounterFact (10,000 edits) | Score: 86.2 -> 91.2 | Helped |
| Use SWEA⊕OS instead of PMET | Llama-2 · CounterFact (10,000 edits) | Score: 83.2 -> 89.6 | Helped |
| Use SWEA⊕OS instead of MEMIT | GPT-J · zsRE (10,000 edits) | Score: 50.2 -> 51.0 | Helped |
| Use SWEA⊕OS instead of PMET | Llama-2 · zsRE (10,000 edits) | Score: 23.9 -> 25.5 | Helped |
| Use SWEA⊕OS instead of ROME | GPT-J · RippleEdits RANDOM | Avg. accuracy: 0.56 -> 0.64 | Helped |
| Use SWEA⊕OS instead of MEMIT | Llama-2 · RippleEdits POPULAR | Avg. accuracy: 0.47 -> 0.43 | Hurt |
| Use SWEA⊕OS instead of MEMIT | GPT-J · 10,000 edits from scratch | Execution time: 28.46 -> 14.86 | Helped |
| Use SWEA⊕OS instead of MEMIT | Llama-2 · 10,000 edits from scratch | Execution time: 33.31 -> 27.46 | Helped |
| Add the suppression step (ablation) | SWEA⊕OS on GPT-J · RippleEdits RANDOM | Avg. accuracy: 0.53 -> 0.64 | Helped |
| Add the suppression step (ablation) | SWEA⊕OS on GPT-J · RippleEdits POPULAR | Avg. accuracy: 0.44 -> 0.54 | Helped |
| Add the suppression step (ablation) | SWEA⊕OS on Llama-2 · RippleEdits RANDOM | Avg. accuracy: 0.58 -> 0.60 | Helped |
| Add the suppression step (ablation) | SWEA⊕OS on Llama-2 · RippleEdits POPULAR | Avg. accuracy: 0.42 -> 0.43 | Helped |
Surprising findings
- On RippleEdits, SWEA⊕OS has poor Relation Specificity (RS), which the authors attribute to unintended knowledge introduced during objective optimization.
- Removing the suppression step degrades GPT-J more than Llama-2 in the RippleEdits ablation, which the authors attribute to Llama-2's larger parameter count.
Limitations
- Only handles unique subjects; aliases are not considered in the current version.
- Poor Relation Specificity on RippleEdits indicates unintended knowledge leakage.
- Token-level matching adds CPU-side inference latency, and edits are cached externally in files.
Experimental setup
Datasets: CounterFact zsRE RippleEdits
Benchmarks: CounterFact zsRE RippleEdits
Metrics: Score Efficacy Generalization Specificity Fluency Consistency LG CI CII SA
Baselines: FT-W MEND MALMEN GRACE ROME MEMIT PMET
Builds on: MEMIT ROME PMET knowledge attribution (integrated gradients)
Closest papers by meaning
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.89)
- Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.88)
- UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models (NeurIPS 2025 · similarity 0.88)
- Spectral Editing of Activations for Large Language Model Alignment (NeurIPS 2024 · similarity 0.87)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.87)
- Should We Really Edit Language Models? On the Evaluation of Edited Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.