Research map / Vision-language and multimodal / CLIP and zero-shot
Approximate Domain Unlearning for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Approximate Domain Unlearning (ADU) for VLMs, which forgets specified image domains while retaining others, using a Domain Disentangling Loss (DDL) and an Instance-wise Prompt Generator (InstaPG) on CLIP vision prompts.
Problem
Pre-trained VLMs generalize across visual domains (real, painting, sketch, etc.) and cannot be selectively made to forget a domain while retaining others, since domain distributions are entangled in feature space; class-level approximate unlearning does not address this.
Approach
Define approximate domain unlearning and explicitly disentangle domain distributions via a domain classifier cross-entropy plus a maximized MMD term (DDL), together with an attention-based instance-wise prompt generator (InstaPG) that produces per-image vision prompts.
Key results
- On Office-Home with one forgotten domain, the method reaches H=69.96 versus 52.59 for the Baseline and 43.44 for CLIPFit.
- On Mini DomainNet with |D_forget|=3, H=81.78 (ablation table) versus 68.82 for the Baseline; forgetting rate reaches 86.12.
- Domain classification accuracy on Office-Home rises from 25.80% before unlearning to 79.43% with the method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ADU with DDL and InstaPG rather than the conventional prompt baseline | CLIP ViT-B/16 · Office-Home, |D_forget|=1 | H: 52.59 -> 69.96 | Helped |
| apply ADU with DDL and InstaPG rather than the conventional prompt baseline | CLIP ViT-B/16 · Mini DomainNet, |D_forget|=1 | H: 62.07 -> 75.56 | Helped |
| remove DDL while retaining InstaPG (ablation) | ADU method · Office-Home, |D_forget|=1 | H: 69.96 -> 56.41 | Hurt |
| remove InstaPG while retaining DDL (ablation) | ADU method · Office-Home, |D_forget|=1 | H: 69.96 -> 60.82 | Hurt |
| remove DDL while retaining InstaPG (ablation) | ADU method · Mini DomainNet, |D_forget|=1 | H: 75.56 -> 64.06 | Hurt |
| remove InstaPG while retaining DDL (ablation) | ADU method · Mini DomainNet, |D_forget|=1 | H: 75.56 -> 74.23 | Hurt |
| add MMD to CE in DDL (ablation) | ADU method · Office-Home, |D_forget|=1 | H: 64.01 -> 69.96 | Helped |
| add CE to MMD in DDL (ablation) | ADU method · Office-Home, |D_forget|=1 | H: 68.62 -> 69.96 | Helped |
| add DDL and InstaPG (ablation) | ADU method · Office-Home, domain classification | accuracy: 31.06 -> 79.43 | Helped |
| use prompt tuning instead of parameter tuning (ablation) | ADU method · Office-Home, |D_forget|=1 | H: 3.90 -> 66.81 | Helped |
| use domain estimation for unlabeled samples (ablation) | ADU method · Office-Home, 70% unlabeled samples | H: 49.97 -> 61.21 | Helped |
| increase the MMD kernel choice from stable RBF to polynomial (ablation) | ADU method · Office-Home, |D_forget|=1 | H: 69.96 -> 36.53 | Hurt |
Surprising findings
- Naive class-unlearning losses applied to domains (Baseline) give only modest forgetting, and the domain-agnostic Baseline's forgetting stays below 40% on Office-Home and Mini DomainNet at one forgotten domain.
- Full parameter tuning alongside prompts causes severe catastrophic forgetting, collapsing Mem to around 2%.
Limitations
- Assumes domain labels are available for all training samples; partial labels require pseudo-labeling and degrade performance.
- Forgetting is weaker for some domains, e.g., real in Office-Home (accuracy falls only to 55.48%), attributed to CLIP pretraining bias.
- Evaluated only with few-shot (8 per domain) training; the autonomous-driving application is a proxy experiment on Mini DomainNet, not real driving data.
Experimental setup
Datasets: ImageNet-1K ImageNet-Sketch Office-Home Mini DomainNet DomainNet
Benchmarks: ImageNet Office-Home Mini DomainNet DomainNet
Metrics: Mem accuracy For error/forgetting rate harmonic mean H
Baselines: LP++ CLIPFit Black-Box Forgetting (BBF) Baseline (L_memorize + L_forget vision prompt) Zero-shot CLIP
Builds on: CoOp deep prompting (VPT/MaPLe-style) maximum mean discrepancy Black-Box Forgetting
Closest papers by meaning
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models (NeurIPS 2024 · similarity 0.88)
- Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · similarity 0.88)
- Explaining Domain Shifts in Language: Concept Erasing for Interpretable Image Classification (CVPR 2025 · similarity 0.88)
- PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.