Research map / Architectures and efficiency / Attention and transformers
PMET: Precise Model Editing in a Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMET, a weight-modified model editing method that jointly optimizes MHSA and FFN hidden states but updates only FFN weights, using a square-root residual spread across critical layers.
Problem
Existing optimization-based model editing methods (ROME, MEMIT) treat the full Transformer layer hidden state as the target value for FFN updates, which mixes in information not required by FFN and reduces edit precision.
Approach
Optimize the Transformer Component (MHSA and FFN) hidden states jointly at the last critical layer, but use only the optimized FFN component hidden state as the value for updating FFN weights, based on the finding that MHSA acts as a general knowledge extractor whose weights need not change; residuals are spread with a square-root schedule across critical layers.
Key results
- On 10K CounterFact edits on GPT-J, PMET reaches score 86.2 vs. MEMIT 85.8, with generalization 92.8 vs. 88.6.
- On 10K zsRE edits on GPT-J, PMET achieves efficacy 96.9, generalization 90.6, specificity 26.7, slightly above MEMIT (96.7, 89.7, 26.6).
- On 10K CounterFact edits on GPT-NeoX (20B), PMET scores 84.3 vs. MEMIT 82.0, but has lower specificity (70.3 vs. 70.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PMET instead of MEMIT | GPT-J (6B) · CounterFact, 10K edits | Score: 85.8 -> 86.2 | Helped |
| Use PMET instead of MEMIT | GPT-NeoX (20B) · CounterFact, 10K edits | Generalization: 82.2 -> 89.4 | Helped |
| Use PMET instead of MEMIT | GPT-J (6B) · zsRE, 10K edits | Generalization: 89.7 -> 90.6 | Helped |
| Add MHSA hidden-state optimization (ablation) | PMET on GPT-J (6B) · CounterFact, 1K–10K edits | Generalization: 96.6 -> 96.1; 95.0 -> 94.5; 89.0 -> 92.8 | Mixed |
| Also update MHSA weights (ablation) | PMET on GPT-J (6B) · CounterFact, 1K–10K edits | Generalization: 96.1 -> 96.8; 94.5 -> 96.0; 92.8 -> 93.5 | Helped |
| Also update MHSA weights (ablation) | PMET on GPT-J (6B) · CounterFact, 1K–10K edits | Specificity: 80.1 -> 78.9; 74.2 -> 70.8; 71.4 -> 68.6 | Hurt |
| Use even residual spread instead of square-root spread (ablation) | PMET on GPT-J (6B) · CounterFact, 1K–10K edits | Generalization: 96.1 -> 86.7; 94.5 -> 82.2; 92.8 -> 78.4 | Hurt |
| Use even residual spread instead of square-root spread (ablation) | PMET on GPT-J (6B) · CounterFact, 1K–10K edits | Specificity: 80.1 -> 82.2; 74.2 -> 79.4; 71.4 -> 78.4 | Helped |
Surprising findings
- Additionally updating MHSA output weights slightly improves generalization but worsens specificity, supporting the view that MHSA should not be edited.
- Even spreading of residuals improves specificity and fluency but substantially lowers efficacy, generalization and consistency compared to square-root spreading.
Limitations
- Edited models cannot reason over edited knowledge (e.g., PM example), so internalization is not achieved.
- Specificity is lower than MEMIT on CounterFact for GPT-J and GPT-NeoX due to square-root residual spreading.
- Benchmarks do not strictly match the subject-centric problem definition.
Experimental setup
Datasets: CounterFact zsRE Wikitext
Benchmarks: CounterFact zsRE
Metrics: score (harmonic mean) efficacy generalization specificity fluency consistency
Baselines: FT-W MEND ROME MEMIT
Builds on: ROME MEMIT MEND Geva et al. FFN key-value memories
Closest papers by meaning
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.87)
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.87)
- On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention for Long-Context LLM Serving (ICML 2025 · similarity 0.86)
- Localizing Knowledge in Diffusion Transformers (NeurIPS 2025 · similarity 0.86)
- Knowledge Circuits in Pretrained Transformers (NeurIPS 2024 · similarity 0.86)
- Transformer Layers as Painters (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.