Research map / Vision-language and multimodal / Multimodal LLMs

SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SALMUBench, a synthetic-persona benchmark for association-level unlearning in CLIP, with Compromised and Clean ViT-B/16 models trained from scratch and holdout sets to detect over-generalization; nine unlearning baselines are evaluated and none avoids all failure modes.

Problem

Machine unlearning for contrastively trained CLIP-style encoders is underexplored, and existing evaluations cannot diagnose fine-grained association-level forgetting or collateral damage from removing sensitive persona-attribute associations.

Approach

Train Compromised and Clean CLIP ViT-B/16 models from scratch on ~400M real pairs (plus 60K synthetic sensitive persona pairs for Compromised only) so the Clean model serves as a retrain-from-scratch reference, and add structured holdout sets (holdout_identity, holdout_association) to measure both forgetting and over-generalization.

Key results

• Compromised model RetFail rises to 0.236 versus 0.001 for the Clean model, while ImageNet zero-shot accuracy stays near 0.64 for both models.
• VLUnlearn, DELETE and CLIPErase reach RetFail 0.001 at 5x budget with GenKnow at or above 0.632, but DELETE and CLIPErase over-generalize with InterIdSim and IntraIdSim around 0.023-0.024 versus a 0.321 target.
• Direct Similarity Minimization reaches RetFail 0.001 but drives AssocStr to -0.429 and InterIdSim to -0.420, indicating embedding-structure damage.

Surprising findings

• Applying unlearning methods to the Clean model, which never saw the sensitive data, still causes utility degradation, e.g., Negative Gradient drives AssocStr to -0.114 and Direct Similarity Minimization to -0.371.

Limitations

• Benchmark focuses on structured PII attributes and CLIP dual encoders rather than diffusion models or implicit sensitive concepts.
• Real retain images were used without face-blurring and without specific consent, a stated ethical concern.
• Hyperparameters are fixed across budgets, so methods were not re-tuned per budget; KS test on 100 real portraits has limited power.

Experimental setup

Datasets: DataComp CommonPool SFHQ Part 4 FHIBE Pantheon

Benchmarks: ImageNet-1K zero-shot COCO retrieval Flickr30K retrieval SALMUBench

Metrics: MRR mean cosine similarity zero-shot top-1 accuracy ACS probe accuracy McNemar / Wilcoxon / KS tests

Baselines: Generic Captions Shuffled Captions Direct Similarity Minimization Negative Gradient Finetuning Descent to Delete VLUnlearn DELETE CLIPErase

Builds on: CLIP TOFU MultiDelete CLIPErase DELETE

Closest papers by meaning

• Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models (NeurIPS 2024 · similarity 0.89)
• Targeted Forgetting of Image Subgroups in CLIP Models (CVPR 2025 · similarity 0.88)
• CoUn: Empowering Machine Unlearning via Contrastive Learning (NeurIPS 2025 · similarity 0.87)
• Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance (NeurIPS 2026 · similarity 0.87)
• From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization (NeurIPS 2025 · similarity 0.87)
• Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset (ICLR 2025 · similarity 0.87)