Research map / Vision-language and multimodal / Multimodal LLMs

ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ComprehendEdit, an 8-task multimodal knowledge editing benchmark built from GQA, TallyQA, VSR, TextVQA and MathVista, plus KGI/KPI in-domain metrics and a two-stage in-context editing baseline (HICE).

Problem

Existing multimodal knowledge editing evaluations (e.g., MMEdit/E-VQA) are narrow in task coverage, rely on AI-generated images and rephrasings that introduce content shifts, and only check out-of-domain locality, missing effects on in-domain samples.

Approach

Build a broad 8-task benchmark from real datasets without synthetic images, add in-domain metrics (KGI and KPI) that measure editing effects on same-source samples, and propose HICE, which classifies whether an input is in the edited scope before applying retrieval-based in-context demonstrations.

Key results

• On the ablation with MiniGPT-4 on E-VQA, HICE reaches Rel 93.16, T-G 90.39, T-L 99.62, M-L 81.58, versus 75.86 Rel and 37.63 M-L for the baseline without memories.
• Adding the challenge-sample memory M2 raises M-L from 52.56 to 77.80 in the ablation, while Rel and T-G drop slightly.
• The authors report that existing methods such as MEND and IKE perform well on Rel/T-G/locality but poorly on KGI and KPI, indicating the bias of earlier evaluations.

Measured effects reported in the paper

Surprising findings

• Methods that excel on conventional metrics (e.g., IKE on Rel and T-G) perform poorly on the proposed in-domain KGI and KPI metrics.
• Fine-tuning the vision module is less effective than fine-tuning the language module.

Limitations

• HICE shows limited gains on KGI and KPI because multimodal questions are tightly tied to the image, and text-only context cannot fully address related questions.
• Benchmark construction filters samples using the same MLLMs (BLIP-2 OPT and MiniGPT-4) that are evaluated, which may bias the difficulty split.
• Multimodal generality is not measured because of content shifts in generated images.

Experimental setup

Datasets: GQA TallyQA VSR TextVQA MathVista E-VQA NQ OK-VQA

Benchmarks: ComprehendEdit E-VQA

Metrics: Reliability Text Generality Text Locality Multimodal Locality I-KGI T-KGI I-KPI T-KPI

Baselines: FT-V FT-L IKE SERAC MEND

Builds on: MMEdit IKE MEND SERAC KEBench

Closest papers by meaning

• MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge (ICLR 2025 · similarity 0.92)
• Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration (NeurIPS 2024 · similarity 0.91)
• UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models (NeurIPS 2025 · similarity 0.90)
• MemEIC: A Step Toward Continual and Compositional Knowledge Editing (NeurIPS 2025 · similarity 0.89)
• VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark (NeurIPS 2024 · similarity 0.89)
• Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.89)