Research map / Vision-language and multimodal / Multimodal LLMs
Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniKE, a unified multimodal knowledge editing framework that represents intrinsic (FFN) and external (in-context, latent) knowledge as key-value memories in the same layers and disentangles knowledge into semantic and truthfulness spaces via contrastive learning.
Problem
Multimodal knowledge editing of MLLMs struggles to jointly achieve reliability, generality, and locality; intrinsic editing lacks generality and external in-context editing lacks locality, and naively combining them yields poor results.
Approach
Unify intrinsic knowledge (extra FFN key-value pairs, assimilation) and external knowledge (latent in-context hidden states as external key-value memory via attention feature shifting, accommodation) at the same semantic level in the same transformer layers, then disentangle knowledge representations into semantic and truthfulness spaces with contrastive learning: a truthfulness editing direction
Key results
- On one-step MMEdit with MiniGPT-4, UniKE reaches average generality 95.1 and locality 93.8 versus MEND's 88.6 and 89.4, while keeping reliability at 98.0.
- In 10-step sequential editing, UniKE averages 90.2 reliability, 86.2 generality and 85.9 locality, outperforming MEND across metrics.
- Naively combining T-Patcher and IKE yields poor generality and locality, while the unified latent paradigm plus disentangling (Row 4 vs Row 3 of ablation) improves VQA locality from 63.8 to 95.2 with Intrin+Latent-IKE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add enhanced knowledge collaboration (semantic/truthfulness disentangling) on top of intrinsic + latent IKE (ablation) | UniKE · E-VQA ablation | M-Locality: 85.3 -> 88.8 | Helped |
| Unify intrinsic and external editing in latent space (Intrin+Latent-IKE vs naive Intrin+IKE) (ablation) | Intrinsic + in-context editing · E-VQA ablation | T-Locality: 63.8 -> 95.2 | Helped |
| Adaptive alpha from semantic similarity vs fixed alpha values (ablation) | Latent in-context knowledge injection · E-VQA analysis | generality and locality | Helped |
| Add truthfulness direction zeta regulating intrinsic editing vs random or semantic zeta (ablation) | Intrinsic knowledge editing · E-VQA analysis | generality and editing performance | Helped |
| Intrinsic-only editing (no external latent IKE) (ablation) | UniKE components · E-VQA ablation | Generality and locality: M-Gen 67.4 vs full 92.8 | Hurt |
| Latent IKE only (no intrinsic editing) (ablation) | UniKE components · E-VQA ablation | T-Locality and M-Locality: T-Loc 54.1, M-Loc 30.5 vs full 98.7/88.8 | Hurt |
| Scale in-context hidden states (more retrieved states) with latent IKE (ablation) | UniKE latent in-context editing · E-VQA | generality and locality | Helped |
| Replace latent-space in-context editing with vanilla IKE that adds more in-context samples (ablation) | IKE · E-VQA | locality | Hurt |
| Unified intrinsic + external editing framework vs proposed method on one-step MMEdit | UniKE vs MEND · Average one-step MMEdit | Locality and Generality: Locality 93.8 vs 89.4; Generality 95.1 vs 88.6 | Helped |
| Intrinsic-only T-Patcher vs UniKE | T-Patcher · Average one-step MMEdit | Generality: 71.6 vs 95.1 (UniKE) | Hurt |
| External-only in-context editing (IKE) vs UniKE | IKE · Average one-step MMEdit | Locality: 26.8 vs 93.8 (UniKE) | Hurt |
| Sequential editing: UniKE vs MEND on 10-step VQA | UniKE · Sequential editing 10-step VQA | Reliability: 91.5 vs 90.6 | Helped |
Surprising findings
- Simply combining intrinsic editing (T-Patcher) with vanilla in-context editing (IKE) gives worse generality and locality than either component alone in some settings, motivating the unified paradigm.
Limitations
- Only applies to MLLMs with visual comprehension; does not address visual generation editing.
- Not evaluated on larger MLLMs (e.g., 65B LLaMA Adapter V2) due to resource limits.
Experimental setup
Datasets: MMEdit
Benchmarks: MMEdit (E-VQA) MMEdit (E-IC)
Metrics: reliability generality locality
Baselines: Fine-tuning (last layer) KE MEND T-Patcher IKE SERAC
Builds on: T-Patcher In-context editing (IKE) MEND SERAC MMEdit
Closest papers by meaning
- ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing (AAAI 2025 · similarity 0.91)
- MemEIC: A Step Toward Continual and Compositional Knowledge Editing (NeurIPS 2025 · similarity 0.90)
- MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge (ICLR 2025 · similarity 0.90)
- Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
- Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs (ICLR 2025 · similarity 0.87)
- Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.