Research map / Large language models / Retrieval and knowledge
Joint Knowledge Editing for Information Enrichment and Probability Promotion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a contrast-based probe that identifies low-layer Information Enrichment and high-layer Probability Promotion stages, and JEEP, a joint locate-then-edit method that modifies MLPs in both regions for large-scale factual and counterfactual knowledge editing.
Problem
Locate-then-edit knowledge editing methods rely on ablation-based probes of original answers, which are inconsistent with the editing goal of changing predictions to target answers; they edit only low layers, leaving critical recall stages unmodified.
Approach
A contrast-based probe compares the probability and rank information flow of original versus target answers across layers, revealing low-layer information enrichment and high-layer probability promotion stages; JEEP jointly edits MLPs in both regions with synergistic optimization (shared objective) and adaptive norm clamping.
Key results
- On Multi-COUNTERFACT with 10,000 edits on LLaMA, JEEP reaches Score 85.9 versus 84.8 for PMET and 84.1 for MEMIT.
- On zsRE with 10,000 edits on LLaMA, JEEP reaches Score 72.3 versus 70.4 for MEMIT.
- On zsRE with 10,000 edits on GPT-J, JEEP reaches Score 51.5 versus 51.0 for PMET.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JEEP instead of PMET | GPT-J (6B) editing on zsRE · zsRE, 10,000 edits | Score: 51.0 -> 51.5 | Helped |
| Use JEEP instead of MEMIT | LLaMA (7B) editing on zsRE · zsRE, 10,000 edits | Score: 70.4 -> 72.3 | Helped |
| Use JEEP instead of PMET | GPT-J (6B) editing on Multi-COUNTERFACT · Multi-COUNTERFACT, 10,000 edits | Score: 86.2 -> 86.5 | Helped |
| Use JEEP instead of PMET | LLaMA (7B) editing on Multi-COUNTERFACT · Multi-COUNTERFACT, 10,000 edits | Score: 84.8 -> 85.9 | Helped |
| Add the low-region update (δ' and Step2) (ablation) | JEEP on LLaMA (7B) · Multi-COUNTERFACT, 10,000 edits | Score: 66.3 -> 85.9 | Helped |
| Add the high-region update (δ* and Step3) (ablation) | JEEP on LLaMA (7B) · Multi-COUNTERFACT, 10,000 edits | Score: 84.1 -> 85.9 | Helped |
| Perform the low-region update Step2 (ablation) | JEEP on LLaMA (7B) · Multi-COUNTERFACT, 10,000 edits | Score: 40.0 -> 85.9 | Helped |
| Perform the high-region update Step3 (ablation) | JEEP on LLaMA (7B) · Multi-COUNTERFACT, 10,000 edits | Score: 81.9 -> 85.9 | Helped |
| Optimize low- and high-region updates synergistically rather than separately (ablation) | JEEP on LLaMA (7B) · Multi-COUNTERFACT, 10,000 edits | Score: 83.3 -> 85.9 | Helped |
| Use adaptive residual spreading instead of even spreading in Step2 (ablation) | JEEP on LLaMA (7B) · Multi-COUNTERFACT, 10,000 edits | Score: 85.4 -> 85.9 | Helped |
| Edit MLP rather than MHSA in Step3 (ablation) | JEEP on LLaMA (7B) · Multi-COUNTERFACT, 10,000 edits | Score: 85.2 -> 85.9 | Helped |
Surprising findings
- Editing only high layers (probability promotion) performs worse than editing only low layers, suggesting enrichment is needed first.
- Upper-most layers act as distribution normalizers that increase stopword probability, so they were left unedited.
Limitations
- Gains over strongest locate-then-edit baselines (MEMIT, PMET) are often small, e.g. around 1 Score point.
- Layer selection and clamp ratios are hand-tuned per model and dataset based on probes.
Experimental setup
Datasets: zsRE Multi-COUNTERFACT Wikipedia
Benchmarks: zsRE Multi-COUNTERFACT
Metrics: Efficacy Success Generalization Success Locality Success Score (harmonic mean)
Baselines: FT-WD MEND MALMEN ROME MEMIT PMET
Builds on: MEMIT PMET ROME MEND MALMEN
Closest papers by meaning
- Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.89)
- Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models Via Layer Gradient Analysis (NeurIPS 2026 · similarity 0.88)
- History Matters: Temporal Knowledge Editing in Large Language Model (AAAI 2024 · similarity 0.88)
- Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models (AAAI 2025 · similarity 0.88)
- Revealing and Mitigating Over-Attention in Knowledge Editing (ICLR 2025 · similarity 0.88)
- Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.