Research map / Large language models / Retrieval and knowledge
Reinforced Lifelong Editing for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLEdit, which reframes hypernetwork-based lifelong model editing as a reinforcement learning problem (editing losses as rewards, offline sequence-level policy optimization) to scale to over 20,000 edits of LLMs.
Problem
Hypernetwork-based model editing fails in lifelong editing (around 100 edits) because the hypernetwork is trained on gradients from a fixed LLM, while the LLM's parameters change during sequential editing.
Approach
Model the hypernetwork as an RL agent whose actions are parameter updates and whose rewards are negative editing losses (efficacy, locality, memory backtracking, and norm regularization); optimize the hypernetwork offline over the full knowledge sequence trajectory with gamma=1 rather than per-edit online updates.
Key results
- On FEVER with Llama-3-8B, 400x20 batch editing, RLEdit reaches 95.34% efficacy and 93.58% generalization versus 94.22% and 94.14% for AlphaEdit, with 0.2238s per edit versus 6.2307s.
- RLEdit is reported to scale beyond 10,000 (and up to over 20,000) sequential edits, where hypernetwork baselines fail after around 100 edits.
- Abstract claims 59.24% average improvement while using 2.11% of the time of most approaches.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RLEdit instead of AlphaEdit | LLaMA-3-8B editing · FEVER | Efficacy: 94.22\std{0.25} -> 95.34\std{0.34} | Helped |
| Use RLEdit instead of DAFNet | LLaMA-3-8B editing · FEVER | Specificity: 66.55\std{0.41} -> 70.36\std{0.29} | Helped |
| Use RLEdit instead of AlphaEdit | Mistral-7B-v0.3 editing · FEVER | Efficacy: 32.74\std{0.42} -> 88.99\std{0.22} | Helped |
| Use RLEdit instead of AlphaEdit | LLaMA-3-8B editing · ZsRE | Efficacy: 86.83\std{0.23} -> 89.42\std{0.34} | Helped |
| Use RLEdit instead of AlphaEdit | Gemma-2-9B editing · ZsRE | Specificity: 30.34\std{0.19} -> 37.15\std{0.41} | Helped |
| Use RLEdit instead of FT | Mistral-7B-v0.3 editing · ZsRE | Specificity: 42.19\std{0.31} -> 27.43\std{0.44} | Hurt |
| Remove RL training framework (ablation) | RLEdit on Llama-3 · CounterFact | Efficacy: 91.75 -> 50.30 | Hurt |
| Remove RL training framework (ablation) | RLEdit on Gemma-2 · ZsRE | Efficacy: 89.22 -> 12.98 | Hurt |
| Remove RL training framework (ablation) | RLEdit on Mistral · FEVER | Generalization: 96.34 -> 45.26 | Hurt |
| Remove memory backtracking (ablation) | RLEdit on Llama-3 · CounterFact | Generalization: 62.40 -> 60.42 | Hurt |
| Remove memory backtracking (ablation) | RLEdit on Gemma-2 · FEVER | Specificity: 71.83 -> 69.28 | Hurt |
| Remove regularization (ablation) | RLEdit on Llama-3 · ZsRE | Generalization: 83.91 -> 83.12 | Hurt |
Surprising findings
- Removing the RL training framework collapses RLEdit to near-zero efficacy on ZsRE (0.62% on Llama-3), indicating the RL formulation is the main driver rather than the architecture.
Limitations
- Evaluation limited to factual knowledge editing datasets (ZsRE, FEVER, CounterFact), not other domains.
- Multi-hop reasoning capability of post-edited LLMs is not explored.
- Performance is notably sensitive to the choice of edited layers.
Experimental setup
Datasets: ZsRE FEVER CounterFact
Benchmarks: ZsRE FEVER CounterFact GLUE (SST, MMLU, MRPC, CoLA, RTE, NLI)
Metrics: Efficacy Generalization Specificity F1 on GLUE tasks editing time per sample
Baselines: FT-L ROME MEMIT MEND MEND* MALMEN MALMEN* DAFNet PRUNE RECT AlphaEdit
Builds on: MEND MALMEN ROME MEMIT AlphaEdit
Closest papers by meaning
- Towards Lifelong Model Editing via Simulating Ideal Editor (ICML 2025 · similarity 0.91)
- MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs (NeurIPS 2025 · similarity 0.91)
- ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA (AAAI 2025 · similarity 0.90)
- Massive Editing for Large Language Models via Meta Learning (ICLR 2024 · similarity 0.89)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.89)
- Perturbation-Restrained Sequential Model Editing (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.