Research map / Architectures and efficiency / Attention and transformers
Mass-Editing Memory in a Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MEMIT instead of FT-W | GPT-J · zsRE, 10,000 edits | Score: 42.1 -> 50.7 | Helped |
| Use MEMIT instead of ROME | GPT-J · CounterFact, 10,000 edits | Editing Score: 50.3 -> 85.8 | Helped |
| Use MEMIT instead of FT-W | GPT-J · CounterFact, 10,000 edits | Editing Score: 67.6 -> 85.8 | Helped |
| Apply MEMIT to 289 star-constellation facts | GPT-J · Amateur astronomy | Accuracy: 53% -> 86% | Helped |
| Spread updates over more critical MLP layers (ablation) | MEMIT · CounterFact | Efficacy and generalization | Helped |
| Edit late-layer MLPs with low causal effect rather than critical mid-layer MLPs (ablation) | MEMIT · CounterFact | Editing performance | Hurt |
| Edit attention modules instead of MLP modules (ablation) | MEMIT · CounterFact | Editing performance | Hurt |
| Increase covariance adjustment factor λ (ablation) | MEMIT · CounterFact | Specificity and fluency | Helped |
| Increase covariance adjustment factor λ (ablation) | MEMIT · CounterFact | Efficacy and generalization | Hurt |
| Mix edits from diverse relations rather than editing each relation separately | MEMIT · CounterFact relation mixes, up to 700 facts | Editing performance | No clear effect |
Closest papers by meaning
- Memory-Augmenting Decoder-Only Language Models through Encoders (Student Abstract) (AAAI 2024 · similarity 0.84)
- MEMORYLLM: Towards Self-Updatable Large Language Models (ICML 2024 · similarity 0.84)
- PMET: Precise Model Editing in a Transformer (AAAI 2024 · similarity 0.84)
- M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.83)
- Interweaving Memories of a Siamese Large Language Model (AAAI 2025 · similarity 0.83)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.