Research map / Vision-language and multimodal / Multimodal LLMs
Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Single Image Unlearning (SIU), which unlearns a concept's visual recognition in MLLMs by fine-tuning on one training image with multifaceted synthetic QA data and a Dual Masked KL-divergence loss plus cross-entropy, and introduces the MMUBench benchmark.
Problem
Machine unlearning of visual recognition of specific concepts (e.g., people, cartoon characters, painting styles) in multimodal LLMs, where only one training image per concept is available and utility must be preserved.
Approach
Builds four types of fine-tuning targets (aligning with unseen concepts via substitute names, new visual descriptions, decoupling factual knowledge, preserving non-targeted knowledge) and a DMK loss that masks concept-specific tokens in the KL term at both token and vocabulary levels, jointly optimized with cross-entropy.
Key results
- SIU on LLaVA-7B reaches 99.0% EM generality on Donald Trump with 60.7 specificity and 61.2 fluency (lower is better), while GA+KL gets 33.0% EM and GA yields 373.6 fluency.
- On LLaVA-13B, SIU keeps 90.0% EM generality versus 10.7-24.7% for baselines.
- Unlearning 20 concepts simultaneously, SIU retains 97.0 EM and 24.9 specificity while GA and GA+KL collapse to repeated tokens with 0 specificity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SIU instead of GA for visual unlearning | LLaVA-7B · Donald Trump | EM: 36.3±5.4 -> 99.0±0.0 | Helped |
| Use SIU instead of PO | LLaVA-7B · Donald Trump | Fluency: 75.1±0.9 -> 61.2±1.2 | Helped |
| Use SIU instead of GA | LLaVA-7B · Donald Trump | C-Dis: 1.6±1.2 -> 1.8±0.3 | Helped |
| Use SIU instead of PO | LLaVA-7B · Donald Trump | Diversity: 93.5±2.1 -> 97.0±0.2 | Helped |
| Use SIU instead of GA | LLaVA-7B · Donald Trump | G-Eval: 1.8±0.4 -> 1.9±0.5 | Hurt |
| Use SIU instead of PO | LLaVA-7B · MIA | ROUGE-L: 0.32 -> 0.27 | Helped |
| Use SIU instead of PO | LLaVA-7B · Multilingual jailbreak | G-Eval: 2.5 -> 2.3 | Helped |
| Use SIU instead of PO | LLaVA-7B · Multi-hop jailbreak | Multi-hop: 0.18 -> 0.16 | Helped |
| Add token-level masking to DMK loss (ablation) | SIU · Donald Trump | EM: 92.0±0.0 -> 99.0±0.0 | Helped |
| Add token-level masking to DMK loss (ablation) | SIU · Donald Trump | C-Dis: 1.5±0.1 -> 1.8±0.4 | Helped |
| Add vocabulary-level masking to DMK loss (ablation) | SIU · Donald Trump | EM: 94.3±1.2 -> 99.0±0.0 | Helped |
| Add vocabulary-level masking to DMK loss (ablation) | SIU · Donald Trump | C-Dis: 1.6±0.2 -> 1.8±0.4 | Helped |
Surprising findings
- Unlearning Donald Trump with SIU also caused the model to misidentify his wife Melania in a group photo, yet it correctly recognized her when cropped alone (positive butterfly effect).
- PO, which trains the model to answer 'I do not know', achieves high EM but low C-Dis (0.4), suggesting it learns a Q&A form rather than truly forgetting.
Limitations
- Evaluated only on LLaVA-7B and 13B and on 20 self-collected concepts, limiting generality of conclusions.
- Relies on GPT-4 for question generation and G-Eval judgments; masking hyperparameters and synthetic data construction require manual effort.
- Specificity is slightly lower than the model without vocabulary-level masking, indicating a utility trade-off.
Experimental setup
Datasets: MMUBench (self-collected, 50 images per concept for 20 concepts, sampled from MIKE concepts) GQA VQA-v2 VisWiz SQA-I VQA-T POPE MMBench MM-Vet
Benchmarks: MMUBench GQA VQA-v2 VisWiz SQA-I VQA-T POPE MMBench MM-Vet
Metrics: Efficacy Exact Match (EM) G-Eval score C-Probability Distance Specificity (average benchmark accuracy) Fluency (perplexity) Diversity (unique words) Min-K% PROB based MIA (Rouge-L) Multilingual jailbreak score Multi-hop jailbreak score
Baselines: PO (preference optimization) GA GA+KL
Builds on: TOFU gradient ascent unlearning Min-K% PROB MIKE dataset
Closest papers by meaning
- AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models (AAAI 2026 · similarity 0.92)
- Machine Unlearning for Image-to-Image Generative Models (ICLR 2024 · similarity 0.90)
- On Effects of Steering Latent Representation for Large Language Model Unlearning (AAAI 2025 · similarity 0.90)
- LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty (CVPR 2025 · similarity 0.90)
- MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs (ICML 2026 · similarity 0.89)
- Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.