Research map / Vision-language and multimodal / Multimodal LLMs
MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMKE-Bench, a multimodal knowledge editing benchmark with natural-language knowledge representation covering visual entity, visual semantic, and user-specific editing, and evaluates several LMM editing methods on it.
Problem
Existing multimodal knowledge editing benchmarks focus on entity-level triplet knowledge and fail to capture complex visual semantics (gestures, emotions, actions) and user-specific knowledge.
Approach
Represents edited knowledge as flexible natural-language descriptions (sourced from Wikipedia and counterfactually edited by LLMs), pairs them with images, and uses text-only reliability questions distinct from training questions, with human verification in construction.
Key results
- IKE achieves best or second-best reliability and generality on MMKE-Bench, but its portability gains over FT-LLM are limited on BLIP2 and MiniGPT4 compared with VLKEB.
- KE achieves the best portability in five of nine tasks, the most of any method.
- FT-Alignment and SERAC keep T-Loc near 100 on original tests, but drop-free behavior weakens with harder LLM-generated T-Loc questions (e.g., FT-LLM falls from 79.62 to 59.21 on LLaVA-1.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use harder LLM-generated T-LOC questions instead of original ones (ablation) | FT-LLM, IKE, KE, MEND (editing methods on LLaVA-1.5) · visual semantic editing T-Loc | T-Loc: FT-LLM 79.62→59.21; IKE 61.10→50.74; KE 71.39→63.53; MEND 98.15→95.65 | Hurt |
| Use harder LLM-generated T-LOC questions instead of original ones (ablation) | FT-Alignment and SERAC · visual semantic editing T-Loc | T-Loc: FT-Alignment 100.00→100.00; SERAC 99.99→99.97 | No clear effect |
| Increase editing-to-test gap in user-specific editing (visual semantic dataset) (ablation) | FT-LLM and FT-Alignment · user-specific editing, visual semantic | performance | Hurt |
| Increase editing-to-test gap in user-specific editing (visual semantic dataset) (ablation) | SERAC · user-specific editing, visual semantic | performance | No clear effect |
| Use the MMKE-Bench benchmark (natural-language, multi-fact descriptions with separate training and evaluation questions) instead of prior triplet benchmarks | MMEdit evaluation · MMKE-Bench vs prior benchmarks | I-Gen / editing performance | No clear effect |
| Use parameter-based KE rather than other methods for portability | KE · portability across Tables 3-5 | portability | Helped |
| Replace LLM-generated evaluation questions with human-verified and rewritten questions | evaluation question generation · benchmark construction quality | successful rate of questions: 81 of 100 LLM-generated questions were good before verification | Helped |
Surprising findings
- Harder LLM-generated locality questions drop most methods' T-Loc, while FT-Alignment and SERAC stay nearly unchanged.
- LLM-generated evaluation questions had only about 81% acceptable quality before human rewriting.
Limitations
- Benchmark uses counterfactual synthetic data rather than real-world knowledge updates.
- Training data size is modest, which may limit methods like MEND and SERAC.
- User-specific editing practicality versus memory-based retrieval is debated.
Experimental setup
Datasets: VLKEB MC-MKE MMEdit
Benchmarks: MMKE-Bench VLKEB MMEdit
Metrics: reliability generality portability locality (T-Loc) I-Gen T-Rel
Baselines: FT-LLM FT-Alignment IKE SERAC MEND KE
Builds on: MMEdit VLKEB MEND SERAC IKE
Closest papers by meaning
- ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing (AAAI 2025 · similarity 0.92)
- VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark (NeurIPS 2024 · similarity 0.92)
- Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration (NeurIPS 2024 · similarity 0.90)
- MemEIC: A Step Toward Continual and Compositional Knowledge Editing (NeurIPS 2025 · similarity 0.89)
- Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit (AAAI 2025 · similarity 0.89)
- Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.