Research map / Vision-language and multimodal / Multimodal LLMs

Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Attribution analysis of visual representation contributions in VLLMs motivates VisEdit, a visual edit adapter with an influence mapper that edits mid-to-late visual hidden states for knowledge correction.

Problem

Model editing for vision-language models (VLLMs) has been little studied, and LLM editors transfer poorly because it is unclear how visual representations affect next-token predictions in the decoder.

Approach

Combine contribution allocation (module outputs mapped to vocabulary logits) with noise-perturbation attribution to find that deep-layer attention draws on prompt-relevant visual regions; then insert a cross-attention visual edit adapter at a high-contribution layer, gated by an influence mapper trained with attribution-derived targets.

Key results

• On E-VQA with BLIP2-OPT, VisEdit reaches 97.53 average vs 94.13 for LTE, the strongest baseline.
• On E-VQA with LLaVA-V1.5, VisEdit averages 96.12 vs 89.23 for LTE and 89.60 for MEND.
• On E-IC with LLaVA-V1.5, VisEdit averages 95.94 vs 90.24 for LTE.

Measured effects reported in the paper

Surprising findings

• Vision-only editing baseline FT-V and VisEdit both keep text locality at 100 because only visual components are edited.
• Ablation shows removing the cross-attention module drops average from 96.12 to 58.74, so it is the key component.

Limitations

• Evaluated only on two editing benchmarks (E-VQA, E-IC) and three VLLM backbones.
• Editing is single-edit; retrieval-based multi-edit scenarios are not explored.
• Editing VLLMs by only modifying visual representations may be limited for text-driven knowledge errors.

Experimental setup

Datasets: E-VQA E-IC VQA-v2 COCO Caption OK-VQA NQ

Benchmarks: E-VQA E-IC

Metrics: reliability text generality modal generality text locality modal locality average

Baselines: FT-V FT-L KE IKE SERAC MEND TP LTE

Builds on: ROME MEMIT LTE SERAC MEND

Closest papers by meaning

• VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark (NeurIPS 2024 · similarity 0.92)
• Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts (CVPR 2025 · similarity 0.91)
• MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge (ICLR 2025 · similarity 0.89)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.89)
• VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models (CVPR 2026 · similarity 0.89)