Research map / Large language models / Retrieval and knowledge
MELO: Enhancing Model Editing with Neuron-Indexed Dynamic LoRA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MELO, a plug-in sequential model editor that trains non-overlapping dynamic LoRA blocks per edit batch and activates them via a neuron-indexed vector database of clustered hidden-state keys.
Problem
Model editing for LLMs must satisfy edit success, locality, generality, sequential editing and efficiency simultaneously, but prior editors either need extra training data or are computationally heavy.
Approach
Train each batch of edits in a non-overlapping partial-rank block of LoRA matrices, and index those blocks with a clustered vector database of last-hidden-state keys so that only the matching LoRA block is activated at inference (zero update for out-of-scope inputs).
Key results
- On zsRE with T5-Small, MELO reaches 0.98 ES, 0.97 generality and 0.72 locality, versus 0.96 ES for GRACE.
- On the Hallucination task with GPT2-XL, MELO attains PPL of 1.04 ES and 2.66 ARR, while GRACE has locality PPL 15.84 vs MELO 17.45.
- MELO edits at 2464 edits/min on T5-Small versus 47.55 for GRACE, with 0.12M extra parameters versus 0.51M for GRACE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MELO instead of GRACE | BERT · SCOTUS | Locality: 0.81 -> 0.96 | Helped |
| Use MELO instead of GRACE | BERT · SCOTUS | Edit Success (ACC): 0.82 -> 0.92 | Helped |
| Use MELO instead of GRACE | T5-Small · zsRE | Locality: 0.69 -> 0.72 | Helped |
| Use MELO instead of GRACE | T5-Small · zsRE | Edit Success (F1): 0.96 -> 0.98 | Helped |
| Use MELO instead of GRACE | T5-Small · zsRE | Generality: 0.94 -> 0.97 | Helped |
| Use MELO instead of GRACE | GPT2-XL · Hallucination | Locality (PPL): 15.84 -> 17.45 | Hurt |
| Use MELO instead of GRACE | GPT2-XL · Hallucination | Edit Success (PPL): 7.14 -> 1.04 | Helped |
| Increase the initial cluster radius (ablation) | MELO · zsRE | Forgotten edits | Hurt |
| Increase the partial rank of dynamic LoRA (ablation) | MELO · zsRE | Edit Success | Helped |
| Increase the partial rank of dynamic LoRA (ablation) | MELO · zsRE | Generality | Helped |
| Vary the partial rank of dynamic LoRA (ablation) | MELO · zsRE | Locality | No clear effect |
| Use keys from the fourth layer rather than other tested layers (ablation) | MELO · zsRE | Edit Success | Helped |
Surprising findings
- Locality of MELO stays the same across different partial ranks because out-of-scope inputs activate no LoRA block.
Limitations
- Locality on the Hallucination task is slightly worse than GRACE (17.45 vs 15.84 PPL).
- Larger cluster radius reduces cluster count but causes more cluster conflicts and forgotten edits.
- Evaluated only on text tasks and on relatively small backbones (BERT, T5, GPT2-XL).
Experimental setup
Datasets: SCOTUS zsRE Natural Questions (NQ) Hallucination (GPT-3 generated biographies) WebText
Benchmarks: SCOTUS zsRE Hallucination
Metrics: accuracy F1 perplexity edit success locality generality ARR editing speed (edits/min) trainable parameters
Baselines: LoRA MEND SERAC CMR ROME GRACE
Builds on: LoRA DyLoRA GRACE T-Patcher
Closest papers by meaning
- Editing Memories Through Few Targeted Neurons (AAAI 2025 · similarity 0.88)
- ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA (AAAI 2025 · similarity 0.87)
- Massive Editing for Large Language Models via Meta Learning (ICLR 2024 · similarity 0.87)
- Should We Really Edit Language Models? On the Evaluation of Edited Language Models (NeurIPS 2024 · similarity 0.87)
- Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs (NeurIPS 2025 · similarity 0.87)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.