Research map / Large language models / Retrieval and knowledge
WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WikiBigEdit, a 500K-pair lifelong knowledge-editing benchmark built from automatically extracted Wikidata snapshot diffs, and compares ROME, MEMIT, WISE, RAG, and LoRA finetuning with merging over eight timesteps.
Problem
Existing knowledge-editing benchmarks are small, synthetic, or predate modern LLM knowledge cutoffs, so it is unclear whether lifelong editing methods work at practical scales of real-world factual updates.
Approach
An automated pipeline that diffs Wikidata snapshots to extract real factual changes and turns them into QA edits with locality, multi-hop, rephrase, and persona evaluation sets, extendable over time; it then benchmarks editing methods against retrieval augmentation and continual LoRA finetuning with merging.
Key results
- RAG reaches about 96-99% update accuracy versus about 41-50% pre-update, and 78.64% rephrase and 66.16% persona accuracy on average, outperforming all editing methods.
- Local editing methods ROME, R-ROME and MEMIT collapse within a few hundred sequential edits, and WISE converges back to pre-update accuracy over 10K edits.
- LoRA finetuning with interpolation merging (weight 0.25) gives consistent gains across all axes and beats plain continual LoRA finetuning after about 100K edits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use retrieval augmentation with stored update question-answer pairs | Llama-2-7B · WikiBigEdit Update | accuracy: 49.90 ($\pm$2.30) -> 96.17 ($\pm$0.60) | Helped |
| Use retrieval augmentation with stored update question-answer pairs | Llama-2-7B · WikiBigEdit Rephrase | accuracy: 47.74 ($\pm$3.65) -> 84.92 ($\pm$5.76) | Helped |
| Use retrieval augmentation with stored update question-answer pairs | Llama-2-7B · WikiBigEdit Personas | accuracy: 41.18 ($\pm$2.26) -> 72.62 ($\pm$2.43) | Helped |
| Use retrieval augmentation with stored update question-answer pairs | Llama-2-7B · WikiBigEdit Mhop | accuracy: 43.91 ($\pm$6.02) -> 51.08 ($\pm$6.66) | Helped |
| Use retrieval augmentation with stored update question-answer pairs | Llama-2-7B · WikiBigEdit Locality | accuracy: 48.62 ($\pm$2.41) -> 56.29 ($\pm$5.09) | Helped |
| Use simple interpolation-based merging of continual LoRA adapters | Continual LoRA finetuning on WikiBigEdit · WikiBigEdit | accuracy across update, rephrase, personas, multi-hop, and locality | Helped |
| Use simple interpolation-based merging of continual LoRA adapters | Continual LoRA finetuning on WikiBigEdit · WikiBigEdit previous timesteps | accuracy | Helped |
| Use RAG rather than knowledge editing | Lifelong knowledge updating across the evaluated LLMs · WikiBigEdit update edits | accuracy | Helped |
| Use RAG for multi-hop reasoning over edited facts | Lifelong knowledge updating across the evaluated LLMs · WikiBigEdit Mhop | accuracy | No clear effect |
| Continue sequentially applying local weight edits with ROME, R-ROME, or MEMIT | Llama-2-7B · WikiBigEdit first 10K updates | update accuracy | Hurt |
| Use WISE for sequential lifelong edits | Llama-2-7B · WikiBigEdit first 10K updates | update accuracy | Hurt |
| Use continual rank-4 LoRA finetuning (ablation) | Mistral-7B and other evaluated models · WikiBigEdit | accuracy | Mixed |
Surprising findings
- Simple continual LoRA finetuning with model merging outperforms specialized knowledge editing methods like WISE and MEMIT at scale.
- RAG still shows forgetting on older timestep generalization data, and multi-hop accuracy stays near pre-update levels because retrieval of both hops is below 50% for the first hop.
Limitations
- Multi-hop reasoning remains weak for all methods, with retrieval of both hops succeeding for only about 10% of samples.
- Evaluation relies on GPT-generated QA pairs, and the Wikidata-based edits are a proxy for real-world knowledge change.
- High-rank LoRA configurations were unstable, with catastrophic failure on Mistral.
Experimental setup
Datasets: Wikidata snapshots (Feb-Jul 2024) WikiBigEdit
Benchmarks: WikiBigEdit
Metrics: accuracy on update edits rephrase accuracy persona accuracy multi-hop accuracy locality
Baselines: ROME R-ROME MEMIT WISE RAG LoRA-FT
Builds on: ROME MEMIT WISE EasyEdit TemporalWiki
Closest papers by meaning
- MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs (NeurIPS 2025 · similarity 0.90)
- UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models (NeurIPS 2025 · similarity 0.89)
- Unveiling the Pitfalls of Knowledge Editing for Large Language Models (ICLR 2024 · similarity 0.89)
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.89)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.89)
- Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.