Research map / Continual learning / Continual fine-tuning of LLMs
Targeted Forgetting of Image Subgroups in CLIP Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a three-stage (forgetting, reminding, restoring) method that selectively unlearns image subgroups within a class in CLIP without pre-training data, using relative Fisher layer selection, BN-statistics-aligned retain set, EMA, and model souping.
Problem
Selectively forgetting fine-grained image subgroups within a class in CLIP without access to pre-training or retain data, while preserving zero-shot performance on other subgroups and datasets.
Approach
Layer selection via relative Fisher information computed on both forget and manually built retain sets, followed by fine-tuning on forget data (LoRA), reminding fine-tuning on retain data with BN-statistic-aligned perturbations and EMA, and model-soup interpolation with the original CLIP calibrated on a small set to restore zero-shot ability.
Key results
- On ImageNet Breeds in-domain forgetting, the proposed method drives target accuracy to 0.0 while keeping overall score 91.0 (RN50) versus 55.6 for the best baseline EMMN.
- On CIFAR-10 airplane forgetting, the method reaches 7.6% target accuracy (random level) and retains at least 66.1% of original zero-shot performance.
- In style forgetting, the method achieves a score of 87.2, versus 46.4 for EMMN and 83.2 for GA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the three-stage forgetting, reminding, and restoring method | CLIP · ImageNet-1K, marmoset subgroup forgetting | Score: 55.6 -> 91.0 | Helped |
| Use the three-stage forgetting, reminding, and restoring method | CLIP · ImageNet-1K, box turtle subgroup forgetting | Score: 59.2 -> 85.9 | Helped |
| Use the three-stage forgetting, reminding, and restoring method | CLIP · CIFAR-10, airplane forgetting | Score: 51.1 -> 88.9 | Helped |
| Use the three-stage forgetting, reminding, and restoring method | CLIP · CIFAR-10, truck forgetting | Score: 61.8 -> 89.5 | Helped |
| Use the three-stage forgetting, reminding, and restoring method | CLIP · ImageNet-1K, sketch-style auto-cars forgetting | Score: 83.2 -> 87.2 | Helped |
| Increase reminding-stage steps from 5 to 20 (ablation) | CLIP · CIFAR-10 retain set | Accr: 59.8 -> 65.6 | Helped |
| Increase reminding-stage steps from 5 to 100 (ablation) | CLIP · ImageNet-1K | Accin: 44.9 -> 0.8 | Hurt |
| Increase reminding-stage steps from 20 to 100 (ablation) | CLIP · CIFAR-10 retain set | Accr: 65.6 -> 32.5 | Hurt |
| Set model-merging coefficient α to 0.9 (ablation) | CLIP · CIFAR-10 forgetting set | Accf: 0.0 -> 68.8 | Hurt |
| Increase model-merging coefficient α from 0.1 to 0.9 (ablation) | CLIP · CIFAR-10 retain set | Accr: 22.7 -> 78.7 | Helped |
| Increase model-merging coefficient α from 0.1 to 0.9 (ablation) | CLIP · CIFAR-10 forgetting set | Accf: 0.0 -> 68.8 | Hurt |
Surprising findings
- Gradient ascent and Fisher/LIP unlearning baselines collapse zero-shot performance under the CLIP unlearning setting, with Fisher and LIP scores near 19-20.
- Over-forgetting of similar subgroups occurs when forgetting only on the target data, motivating the retain set.
Limitations
- Hyperparameters such as merging coefficient alpha and reminding step count are sensitive; large steps cause ImageNet degradation.
- Retain set is manually constructed and coarse labels are used as surrogate descriptions.
- Evaluated only on CLIP RN50 and RN101 with a limited set of forget subgroups.
Experimental setup
Datasets: ImageNet-1K (Breeds) CIFAR-10 Stable Diffusion styled ImageNet-1K
Benchmarks: ImageNet-1K CIFAR-10 Food STL-10 ObjectNet
Metrics: zero-shot accuracy forgetting accuracy on target subgroup retention ratio score
Baselines: FT GA Fisher LIP EMMN CLIP-LIP
Builds on: Fisher forgetting gradient ascent unlearning EMMN LIP CLIP-LIP
Closest papers by meaning
- SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning (CVPR 2026 · similarity 0.88)
- Machine Unlearning for Image-to-Image Generative Models (ICLR 2024 · similarity 0.88)
- Captured by Captions: On Memorization and its Mitigation in CLIP Models (ICLR 2025 · similarity 0.88)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.88)
- Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models (ICLR 2024 · similarity 0.88)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.