Research map / Large language models / Retrieval and knowledge
UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UniEdit, a 311K-sample open-domain LLM knowledge editing benchmark built from Wikidata with a multi-hop chain sampling algorithm (NMCS), and benchmarks six editors on GPT2-XL, GPT-J, and Llama-3.1-8B.
Problem
Existing LLM knowledge editing benchmarks cover narrow knowledge domains and a limited set of generality and locality criteria, so they fail to capture the diversity of ripple effects from edits.
Approach
Build edit, generality and locality samples by sampling multi-hop neighborhood chains from Wikidata across 25 domains (NMCS algorithm), which unifies many prior generality and locality criteria in one benchmark, then verbalize the chains with DeepSeek-V3.
Key results
- SERAC achieves the best average score on GPT2-XL (88.77), GPT-J (89.02) and Llama-3.1-8B (88.96) on UniEdit.
- L&E methods such as ROME and AlphaEdit reach high reliability but low generality (e.g., ROME generality 35.84 on GPT2-XL, 51.38 on Llama-3.1-8B).
- GRACE attains near-perfect locality (99.97-99.99) but low generality (28.00-51.89).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply FT instead of leaving the model unedited | GPT2-XL · UniEdit | reliability: 29.69 -> 100.0 | Helped |
| Apply FT instead of leaving the model unedited | GPT-J · UniEdit | reliability: 35.34 -> 100.0 | Helped |
| Apply FT instead of leaving the model unedited | LLaMa-3.1 · UniEdit | reliability: 43.68 -> 100.0 | Helped |
| Use SERAC rather than the unedited model | GPT2-XL · UniEdit | generality: 28.04 -> 78.79 | Helped |
| Use SERAC rather than the unedited model | GPT-J · UniEdit | generality: 33.04 -> 81.32 | Helped |
| Use IKE rather than the unedited model | LLaMa-3.1 · UniEdit | generality: 51.81 -> 89.52 | Helped |
| Use GRACE rather than the unedited model | GPT2-XL · UniEdit | locality: 100.0 -> 99.99 | Hurt |
| Use GRACE rather than the unedited model | GPT-J · UniEdit | locality: 100.0 -> 99.97 | Hurt |
| Use GRACE rather than the unedited model | LLaMa-3.1 · UniEdit | locality: 100.0 -> 99.97 | Hurt |
| Use SERAC rather than the unedited model | GPT2-XL · UniEdit | average: 52.58$_{\pm0.05}$ -> 88.77$_{\pm0.10}$ | Helped |
| Use SERAC rather than the unedited model | GPT-J · UniEdit | average: 56.13$_{\pm0.03}$ -> 89.02$_{\pm0.17}$ | Helped |
| Use SERAC rather than the unedited model | LLaMa-3.1 · UniEdit | average: 65.16$_{\pm0.02}$ -> 88.96$_{\pm0.08}$ | Helped |
Surprising findings
- Fine-tuning reaches 100% reliability on all backbones yet has only moderate generality, indicating overfitting to individual samples.
- Adding complexity to locality criteria often does not hurt locality, and can even improve it, while generality decreases.
Limitations
- Focuses only on English and does not cover other languages.
- Only single-modality text editing; no vision-language editing evaluation.
- Wikidata source errors and attribute-richness imbalance may propagate into the benchmark.
Experimental setup
Datasets: Wikidata
Benchmarks: UniEdit
Metrics: reliability generality locality average
Baselines: FT ROME AlphaEdit SERAC T-Patcher GRACE IKE
Builds on: ROME MEMIT AlphaEdit SERAC GRACE
Closest papers by meaning
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.93)
- Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.92)
- Unveiling the Pitfalls of Knowledge Editing for Large Language Models (ICLR 2024 · similarity 0.91)
- AnyEdit: Edit Any Knowledge Encoded in Language Models (ICML 2025 · similarity 0.91)
- Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing (ICML 2025 · similarity 0.91)
- Should We Really Edit Language Models? On the Evaluation of Edited Language Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.