Research map / Vision-language and multimodal / Multimodal LLMs
Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FIUBench, a VLM unlearning benchmark built on a fictitious facial identity VQA dataset (400 SFHQ faces, 8,000 QA pairs) with a learn-then-unlearn pipeline, and evaluates four baseline unlearning methods with MIA and adversarial paraphrase attacks.
Problem
Machine unlearning for vision language models under the Right to be Forgotten lacks a benchmark where the unlearning targets are private facial-identity information and where forget quality is robustly evaluated against privacy attacks.
Approach
Fine-tune VLMs on synthetic faces paired with fictitious private profiles and GPT-4o-generated QA pairs so the private knowledge is controlled, then unlearn a forget split and measure forgetting with KS-test, exact match, membership inference (Min-K% Prob) and paraphrase-based adversarial extraction, combined into a weighted forget score.
Key results
- Gradient Ascent on LLaVA-Phi-mini-3B reaches forget score 92.9 (near Retain 93.3) but drops model utility average to 42.5 (vs 88.6 for Retain).
- PO on LLaVA-Phi-mini-3B achieves a low EM of 6.90 (vs 13.3 retain) but MIA stays at 64.2 (vs 12.3 retain), showing private knowledge persists.
- KL retains the highest utility on LLaVA-Phi (78.2 average) but has forget quality of only 36.8, indicating weak forgetting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PO instead of other unlearning approaches | LLaVA-Phi-mini-3B · FIUBench forget set | EM: 41.2 -> 6.90 | Helped |
| Use PO instead of other unlearning approaches | LLaVA-Phi-mini-3B · FIUBench forget set | MIA: 42.1 -> 64.2 | Hurt |
| Use GA or GD gradient ascent-based unlearning | VLMs evaluated on FIUBench · FIUBench forget/retain split | Model utility | Hurt |
| Increase the number of GA, GD, or KL unlearning steps | LLaVA-Phi · FIUBench | Model Utility | Hurt |
| Increase the forget-set size from 1% to 5% or 10% | VLM unlearning methods · FIUBench forget-set splits | Forget quality | Helped |
| Fine-tune the vision encoder and projector instead of only the projector and LLM (ablation) | LLaVA-Phi-Mini-3B with GD · FIUBench | Model Utility: 61.6 -> 58.1 | Hurt |
| Fine-tune only the projector instead of the projector and LLM (ablation) | LLaVA-Phi-Mini-3B with GD · FIUBench | MIA: 13.0 -> 12.6 | Helped |
| Fine-tune only the LLM instead of the projector and LLM (ablation) | LLaVA-Phi-Mini-3B with GD · FIUBench | Model Utility: 61.6 -> 64.8 | Helped |
| Replace VQA-pair-based unlearning with knowledge-level descriptions (KU) (ablation) | LLaVA-Phi-Mini-3B · FIUBench | MIA: 13.0 -> 50.1 | Hurt |
| Replace VQA-pair-based unlearning with knowledge-level descriptions (KU) (ablation) | LLaVA-Phi-Mini-3B · FIUBench | Model Utility: 61.6 -> 73.5 | Helped |
Surprising findings
- Preference optimization looks strong on exact match and APE but MIA reveals private knowledge remains, so refusal-based unlearning does not truly remove knowledge.
- Knowledge-level unlearning (KU) preserves utility well (73.5) but yields weak forget quality (MIA 50.1).
Limitations
- Simulates rare knowledge by fine-tuning on fictitious identities before unlearning; does not cover pretraining-acquired or in-context knowledge.
- Forget quality score combines KS-test, EM, MIA and APE with hand-chosen weights.
- Only two VLMs and four baseline unlearning methods evaluated.
Experimental setup
Datasets: SFHQ Fictitious Facial Identity VQA dataset MME POPE
Benchmarks: MME POPE FIUBench
Metrics: ROUGE-L GPT-Eval score truth ratio KS-test exact match Min-K% Prob MIA adversarial privacy extraction
Baselines: Gradient Ascent Gradient Difference KL Minimization Preference Optimization
Builds on: TOFU LLaVA Min-K% Prob SFHQ
Closest papers by meaning
- Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models (NeurIPS 2024 · similarity 0.89)
- MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models (ICML 2026 · similarity 0.87)
- Generative Unlearning for Any Identity (CVPR 2024 · similarity 0.87)
- AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models (AAAI 2026 · similarity 0.87)
- SineProject: Machine Unlearning for Stable Vision-Language Alignment (CVPR 2026 · similarity 0.87)
- Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.