Research map / Large language models / Retrieval and knowledge
AnyEdit: Edit Any Knowledge Encoded in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnyEdit, an autoregressive model-editing paradigm that splits long-form knowledge into chunks and iteratively edits the last token of each chunk, plus a new EditEverything benchmark for long, diverse-format knowledge.
Problem
Existing locate-then-edit model editing methods rely on single-token hidden-state edits and fail on long-form, diverse-formatted knowledge (poetry, code, math), which has low original probability and long outputs.
Approach
Decompose the target output into sequential chunks and iteratively locate and edit the last token of each chunk conditioned on the input and previously edited chunks, grounded in the chain rule of mutual information; it also works as a plug-in for existing locate-then-edit methods.
Key results
- On UnKEBench with Llama3-8B-Instruct, AnyEdit reaches 97.76 BERT Score and 92.96 ROUGE-L on original questions, versus 76.21 and 30.49 for MEMIT.
- AnyEdit improves BERT Score by 32% and ROUGE-L by 56% on paraphrased UnKEBench questions for Llama3-8B-Instruct relative to baselines, per the paper's observation.
- Integrating AnyEdit into MEMIT, AlphaEdit and UnKE gives improvements across metrics at an average 24.7% increase in editing time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AnyEdit instead of single-token editing | Llama3-8B-Instruct · UnKEBench Ori. | BERT Score: 98.34±0.15 -> 97.76±0.11 | Hurt |
| Use AnyEdit instead of single-token editing | Llama3-8B-Instruct · AKEW Counterfact Ori. | BERT Score: 98.62±0.14 -> 97.76±0.14 | Hurt |
| Use AnyEdit instead of single-token editing | Llama3-8B-Instruct · AKEW MQUAKE Ori. | BERT Score: 98.33±0.13 -> 96.33±0.21 | Hurt |
| Use AnyEdit instead of single-token editing | Qwen2.5-7B-Instruct · UnKEBench and AKEW | BERT Score | Helped |
| Use AnyEdit for paraphrase questions | Llama3-8B-Instruct · UnKEBench Para. | BERT Score: +32% | Helped |
| Use AnyEdit for paraphrase questions | Llama3-8B-Instruct · UnKEBench Para. | ROUGE-L: +56% | Helped |
| Use AnyEdit for diverse-formatted knowledge editing | model editing methods · EditEverything Code | ROUGE-L: +60.58% | Helped |
| Use AnyEdit for diverse-formatted knowledge editing | model editing methods · EditEverything News | ROUGE-L: +52.38% | Helped |
| Add autoregressive AnyEdit editing to existing methods | MEMIT, AlphaEdit, and UnKE · UnKEBench | editing performance | Helped |
| Add autoregressive AnyEdit editing to existing methods | MEMIT, AlphaEdit, and UnKE · Counterfact, MQUAKE, and UnKEBench | editing time: 24.7% | Hurt |
| Increase chunk size beyond a certain threshold (ablation) | AnyEdit · long-form knowledge editing | editing performance | Hurt |
| Replace the AnyEdit update with full-layer gradient descent (AnyEdit*) | Llama3-8B-Instruct · UnKEBench and AKEW | editing performance | Helped |
Surprising findings
- Single-token editing efficacy drops for low-original-probability diverse-formatted knowledge and for long outputs, which the authors frame as a theoretical efficacy barrier.
Limitations
- Not explicitly optimized for lifelong editing scenarios.
- Confined to textual knowledge; no multimodal knowledge support.
- Adds editing time overhead relative to the base methods.
Experimental setup
Datasets: UnKEBench AKEW Orca-Math MBPP Camel-Chemistry Wikipedia
Benchmarks: UnKEBench AKEW (Counterfact) AKEW (MQUAKE) EditEverything
Metrics: BERT Score ROUGE-L editing time per sample
Baselines: FT-L MEND ROME MEMIT AlphaEdit UnKE
Builds on: ROME MEMIT AlphaEdit UnKE
Closest papers by meaning
- AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.92)
- UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models (NeurIPS 2025 · similarity 0.91)
- Should We Really Edit Language Models? On the Evaluation of Edited Language Models (NeurIPS 2024 · similarity 0.90)
- Unveiling the Pitfalls of Knowledge Editing for Large Language Models (ICLR 2024 · similarity 0.89)
- Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing (ICML 2025 · similarity 0.89)
- Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.