Research map / Vision-language and multimodal / Multimodal LLMs

MultiMedBench: A Scenario-Aware Benchmark for Evaluating Knowledge Editing in Medical VQA

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MultiMedEdit, a benchmark for evaluating knowledge editing (Prompt, LoRA, GRACE, WISE) in medical VQA across understanding and reasoning tasks, with reliability, generality, and locality metrics under single and lifelong editing.

Problem

Knowledge editing for multimodal medical VQA models has received little study; existing medical KE benchmarks are text-only, so there is no way to measure how edits to clinical multimodal models succeed, generalize, and cause side effects.

Approach

A scenario-aware benchmark built from medical VQA sources (PMC-VQA, MedFrameQA, MedXpertQA, OmniMedVQA), filtered for questions a medical MLLM gets wrong, split into understanding and reasoning tiers, with a three-part metric suite (reliability, generality, locality) and evaluation of four KE paradigms.

Key results

• Prompt editing achieves high reliability and generality (e.g., 0.9749 reliability on LLaVA-OneVision understanding) but poor locality (around 0.75-0.80 on text/visual locality).
• LoRA performs inconsistently across models, reaching 0.9033 reliability on HuatuoGPT-7B but only 0.1852 on Qwen2-VL understanding.
• GRACE reaches strong reliability but poor generality (e.g., 0.4295 on LLaVA-OneVision understanding), and WISE, GRACE, and LoRA achieve near-perfect locality (1.0).

Surprising findings

• Weight-modifying methods (WISE, GRACE, LoRA) achieve near-perfect locality but show erratic, unpredictable reliability under lifelong editing, with GRACE collapsing after peaking at 750 edits.
• Prompt-based editing has the lowest latency and most stable memory use, despite lower locality.

Limitations

• Focus is limited to question-answering tasks.
• Lacks deeper analysis of interpretability of editing mechanisms.
• Retrieval-augmented editing methods were excluded from the comparison.

Experimental setup

Datasets: PMC-VQA MedFrameQA MedXpertQA OmniMedVQA Natural Questions VQAv2

Benchmarks: MultiMedEdit

Metrics: reliability generality locality edit latency peak GPU memory

Baselines: Prompt LoRA GRACE WISE

Builds on: WISE GRACE LoRA MedEditBench

Closest papers by meaning

• Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.89)
• OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning (CVPR 2026 · similarity 0.88)
• ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing (AAAI 2025 · similarity 0.88)
• Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.88)
• MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.88)
• OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.87)