Research map / Large language models / Retrieval and knowledge
Massive Editing for Large Language Models via Meta Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MALMEN, a hyper-network model editor that aggregates MEND-style parameter shifts via a regularized least-squares normal equation and decouples LM and hyper-network backprop, enabling editing of thousands of facts.
Problem
Hyper-network-based LM editors scale poorly when editing many facts simultaneously: summing parameter shifts causes cancellation effects and meta-training requires prohibitive memory.
Approach
Aggregate per-key parameter shifts into one shift by solving a regularized least-squares problem in closed form (normal equation), and split backpropagation into an LM stage and a hyper-network stage so that batch sizes are arbitrary on both networks.
Key results
- MALMEN edits up to 16,384 facts on GPT-J (6B), more than two orders of magnitude more than MEND with similar performance, and outperforms MEMIT.
- On zsRE with GPT-J at m=8192, MALMEN reaches 99.7% ES and 92.8% GS versus 94.0% ES and 85.5% GS for caching all tokens, with 4.4h vs 7.2h training time.
- Mean residual of the normal-equation aggregation is about three orders of magnitude smaller than summing parameter shifts on GPT-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the normal equation with summing parameter shifts (ablation) | MALMEN · FEVER, m=512, BERT-base; zsRE, m=512, GPT-2 | ES (%): 93.0 -> 79.1; 99.7 -> 52.2 | Hurt |
| replace the normal equation with summing parameter shifts (ablation) | MALMEN · FEVER, m=512, BERT-base; zsRE, m=512, GPT-2 | GS (%): 85.8 -> 75.1; 96.1 -> 50.7 | Hurt |
| replace the normal equation with summing parameter shifts (ablation) | MALMEN · FEVER, m=512, BERT-base; zsRE, m=512, GPT-2 | LS (%): 89.5 -> 72.9; 16.5 -> 17.0 | Mixed |
| remove the regularization term (ablation) | MALMEN · zsRE, m=512, GPT-2 | ES (%): 99.7 -> 0.0 | Hurt |
| remove the regularization term (ablation) | MALMEN · zsRE, m=512, GPT-2 | GS (%): 96.1 -> 0.0 | Hurt |
| remove the regularization term (ablation) | MALMEN · zsRE, m=512, GPT-2 | LS (%): 16.5 -> 0.0 | Hurt |
| edit the first rather than the second FFN linear layer (ablation) | MALMEN · FEVER, m=512, BERT-base; zsRE, m=512, GPT-2 | ES (%): 93.0 -> 50.3; 99.7 -> 97.7 | Mixed |
| edit the first rather than the second FFN linear layer (ablation) | MALMEN · FEVER, m=512, BERT-base; zsRE, m=512, GPT-2 | GS (%): 85.8 -> 50.3; 96.1 -> 89.1 | Hurt |
| edit the first rather than the second FFN linear layer (ablation) | MALMEN · FEVER, m=512, BERT-base; zsRE, m=512, GPT-2 | LS (%): 89.5 -> 51.0; 16.5 -> 15.8 | Mixed |
| cache key and value-gradient tuples for all tokens (ablation) | MALMEN · FEVER, m=512, BERT-base; zsRE, m=512, GPT-2 | ES (%): 93.0 -> 97.0; 99.7 -> 99.2 | Mixed |
| cache key and value-gradient tuples for all tokens (ablation) | MALMEN · FEVER, m=512, BERT-base; zsRE, m=512, GPT-2 | GS (%): 85.8 -> 90.0; 96.1 -> 94.0 | Mixed |
| cache key and value-gradient tuples for all tokens (ablation) | MALMEN · FEVER, m=512, BERT-base; zsRE, m=512, GPT-2 | LS (%): 89.5 -> 90.3; 16.5 -> 16.6 | Mixed |
Surprising findings
- Summing parameter shifts (the MEND approach) achieves strong ES on GPT-2 but the normal equation variant fails to converge without regularization, with 0% success on GPT-2 when the regularization term is removed.
- Caching only tokens that output the answer is better at scale, while caching all tokens is better at small scale on BERT-base.
Limitations
- Meta-gradient computation scales linearly with the number of facts.
- Fails to generalize to rephrasings beyond those in the prompts (no inference of semantically related edits).
- Slightly lower locality than the best baseline on large models (gap under 0.02).
Experimental setup
Datasets: FEVER FEVER2 zsRE Natural Questions
Benchmarks: FEVER zsRE
Metrics: editing success (ES) generalization success (GS) locality success (LS) computation time memory consumption mean residual
Baselines: MEND MEMIT Fine-tuning (FT) Sum param. shifts
Builds on: MEND MEMIT ROME fine-tuning
Closest papers by meaning
- Reinforced Lifelong Editing for Language Models (ICML 2025 · similarity 0.89)
- UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models (NeurIPS 2025 · similarity 0.88)
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.88)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models Via Layer Gradient Analysis (NeurIPS 2026 · similarity 0.88)
- Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.