Research map / Vision-language and multimodal / Multimodal LLMs

MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MLUBench, a 127-entity, 9-class benchmark for lifelong unlearning in MLLMs, shows baselines (GA, GD, KL, NPO) degrade cumulatively, and proposes LUMoE, a LoRA-expert MoE with a MLLM router that keeps forget quality and utility high.

Problem

Removing specific multimodal content from MLLMs arrives as sequential unlearning requests, and existing MLLM unlearning benchmarks are too small or narrow to evaluate the cumulative degradation this causes.

Approach

Build a large real-world entity benchmark split into sequential unlearning tasks, and isolate each task's unlearning in a switchable LoRA expert chosen by an MLLM entity router, leaving the base model untouched for retained queries.

Key results

• On LLaVA-7B, GA's forget quality on Task A drops from 0.38 to 0.01 after subsequent unlearning, showing cumulative degradation.
• LUMoE achieves forget quality of 0.95-1.00 and utility of 0.88-0.93 across tasks on LLaVA-7B.
• Unlearning only the vision side or only the LLM side both cause severe cumulative degradation, and the modality gap grows after unlearning on Qwen3-VL-4B.

Surprising findings

• Unlearning a single modality (vision adapter only or LLM only) is enough to collapse performance, indicating multimodal alignment is the key fragility.
• Replacing the GLM-4V-Plus router with GPT-4o lowers forget quality on Task C to 0.81, showing router choice matters.

Limitations

• LUMoE depends on an external commercial MLLM router (GLM-4V-Plus) for entity extraction, and its performance varies with router choice.
• The forget and retain sets are built from widely known entities, and the forget-quality metric relies on GPT-4o judging rather than a retrained-model reference.
• The benchmark is restricted to 9 entity types from Wikipedia and images from Google Images, which may raise copyright concerns.

Experimental setup

Datasets: MLUBench MLLMU-Bench TruthfulQA MMBench

Benchmarks: MLUBench MLLMU-Bench TruthfulQA MMBench

Metrics: forget quality (GPT rejection score) model utility (GPT correctness score) modality gap

Baselines: GA GD KL NPO MLLMU-Bench

Builds on: Gradient Ascent Negative Preference Optimization Direct Preference Optimization LoRA Mixture of Experts MLLMU-Bench

Closest papers by meaning

• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods (ICLR 2025 · similarity 0.91)
• A Closer Look at Machine Unlearning for Large Language Models (ICLR 2025 · similarity 0.91)
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models (ICLR 2025 · similarity 0.90)
• OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics (NeurIPS 2025 · similarity 0.90)
• Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models (NeurIPS 2024 · similarity 0.89)
• RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models (NeurIPS 2024 · similarity 0.89)