Research map / Large language models / Retrieval and knowledge
ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ELDER, a lifelong model-editing method that routes edits through a learned mixture of LoRAs with a guided allocation loss and a Hamming-distance deferral mechanism to keep general abilities intact.
Problem
Lifelong model editing of LLMs suffers from forgetting of prior edits and from poor robustness to rephrased inputs, because discrete key-value adapter mappings place semantically equivalent inputs on different adapters, and repeated edits degrade general abilities.
Approach
Replace discrete data-to-adapter mappings with a continuous, learned router that combines multiple LoRAs (top-k, sequence-level routing); a guided loss trains the router to reproduce pre-assigned per-edit LoRA allocations so rephrases get similar allocations, and a deferral mechanism compares allocation codes via Hamming distance to bypass the adapters for non-edit inputs.
Key results
- On ZsRE with LLaMA2-7B after 1000 sequential edits, ELDER reaches 93.96 reliability and 90.21 generalization, versus 89.48 and 0.46 for GRACE.
- On CounterFact with GPT2-XL, ELDER reaches 94.65 reliability and 91.26 generalization, with generalization above all baselines except SERAC/MELO-level reliability baselines.
- Test retention averaged over eight general tasks is 31.1 for ELDER versus 31.2 for the pre-edit base model and 31.4 for GRACE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ELDER instead of GRACE | GPT2-XL · ZsRE | reliability: 96.80 -> 97.47 | Helped |
| Use ELDER instead of T-Patcher | GPT2-XL · ZsRE | generalization: 67.74 -> 96.08 | Helped |
| Use ELDER instead of GRACE | LLaMA2-7B · ZsRE | reliability: 89.48 -> 93.96 | Helped |
| Use ELDER instead of ROME | LLaMA2-7B · ZsRE | generalization: 74.53 -> 90.21 | Helped |
| Use ELDER instead of SERAC | GPT2-XL · CounterFact | reliability: 100.00 -> 94.65 | Hurt |
| Use ELDER instead of SERAC | GPT2-XL · CounterFact | generalization: 79.49 -> 91.26 | Helped |
| Use ELDER instead of T-Patcher | LLaMA2-7B · CounterFact | reliability: 88.93 -> 95.07 | Helped |
| Use ELDER instead of ROME | LLaMA2-7B · CounterFact | generalization: 80.30 -> 90.79 | Helped |
| Add guided loss during training (ablation) | ELDER · ZsRE | generalization: 92.49 -> 96.08 (GPT2); 87.19 -> 90.21 (LLaMA2) | Helped |
| Replace guided loss with load-balancing loss (ablation) | ELDER · ZsRE | generalization: 96.08 -> 74.26 (GPT2); 90.21 -> 71.39 (LLaMA2) | Hurt |
| Add guided loss during training (ablation) | ELDER · CounterFact | generalization: 87.57 -> 91.26 (GPT2); 85.20 -> 90.79 (LLaMA2) | Helped |
| Replace guided loss with load-balancing loss (ablation) | ELDER · CounterFact | generalization: 91.26 -> 75.49 (GPT2); 90.79 -> 77.94 (LLaMA2) | Hurt |
Surprising findings
- Load-balancing loss from MoE literature degrades generalization sharply in the small-batch lifelong editing setting (e.g., 74.26 vs 96.08 on ZsRE GPT2-XL).
- GRACE achieves near-zero generalization on ZsRE (0.00 for GPT2-XL) despite high reliability, showing discrete mappings fail on rephrases.
Limitations
- Deferral threshold epsilon and other hyperparameters (k, N, lambda, epsilon) were tuned on the evaluation metrics, risking optimistic results.
- Evaluated only on two models (GPT2-XL and LLaMA2-7B) and two editing datasets with up to 1000 edits in main experiments.
Experimental setup
Datasets: ZsRE CounterFact GSM8K RTE Natural Questions BoolQ MuTual SAMSum CoNLL03 SST2
Benchmarks: ZsRE CounterFact GSM8K RTE Natural Questions BoolQ MuTual SAMSum CoNLL03 SST2
Metrics: reliability generalization test retention on general tasks editing speed number of parameters
Baselines: GRACE MELO SERAC T-Patcher ROME WilKE FT-L LoRA
Builds on: LoRA GRACE MELO Mixture-of-Experts ROME
Closest papers by meaning
- Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts (CVPR 2025 · similarity 0.91)
- MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs (NeurIPS 2025 · similarity 0.91)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.91)
- Towards Lifelong Model Editing via Simulating Ideal Editor (ICML 2025 · similarity 0.90)
- Reinforced Lifelong Editing for Language Models (ICML 2025 · similarity 0.90)
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.