Research map / Vision-language and multimodal / Multimodal LLMs

Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LiveEdit, a lifelong VLLM editor that generates a low-rank expert per edit and routes them with visual hard filtering plus textual soft fusion, evaluated on E-VQA, E-IC, and VLKEB against LLM-adapted editors.

Problem

Lifelong knowledge editing of vision language models (VLLMs) is unexplored; LLM editors do not transfer well because visual inputs add noise and cross-modal interactions, and edits accumulate over many timesteps.

Approach

An editing expert generator produces a low-rank expert per edit sample (stored in a repository), and at inference a two-stage router applies hard filtering on visual key features (with a trainable vision sentinel threshold) followed by soft routing that weights experts by textual prompt similarity with absolute (sigmoid) and relative (softmax) weights.

Key results

• On BLIP2-OPT with 1,000 edits on E-VQA, LiveEdit reaches 93.69 average versus 85.16 for the next best (RECIPE) and 40.96 for FT-L.
• On LLaVA-V1.5 with 1,000 edits on VLKEB, LiveEdit averages 91.79 while LTE and RECIPE reach 68.59 and 69.56.
• LiveEdit keeps near 100% text and modal locality across 1 to 1,000 edits on E-VQA and VLKEB in the reported tables.

Measured effects reported in the paper

Surprising findings

• Locate-then-edit style direct editors (FT-L, FT-M, TP) degrade sharply as edits accumulate, while retrieval-based methods remain more robust.
• Removing the absolute soft-routing loss mainly hurts modal locality, indicating irrelevant hard-routed experts otherwise receive non-trivial weight.

Limitations

• Benchmark construction relies on synthetic image and text rephrasings generated by Stable Diffusion and ChatGLM, which may not reflect real edit distributions.
• Several baselines are adapted from LLM editors to VLLMs with hyperparameters borrowed from original papers and only lightly tuned.

Experimental setup

Datasets: E-VQA E-IC VLKEB

Benchmarks: E-VQA E-IC VLKEB

Metrics: reliability text generality modal generality text locality modal locality average

Baselines: FT-L FT-M MEND SERAC TP LTE RECIPE LEMoE

Builds on: LEMoE RECIPE LTE MEND MMEdit

Closest papers by meaning

• VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark (NeurIPS 2024 · similarity 0.91)
• Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit (AAAI 2025 · similarity 0.91)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA (AAAI 2025 · similarity 0.91)
• MemEIC: A Step Toward Continual and Compositional Knowledge Editing (NeurIPS 2025 · similarity 0.90)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs (NeurIPS 2025 · similarity 0.90)
• WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.90)