Research map / Vision-language and multimodal / CLIP and zero-shot
Captured by Captions: On Memorization and its Mitigation in CLIP Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIPMem, a leave-one-out memorization metric for CLIP that jointly uses both encoders, finds mis-captioned samples are most memorized and text encoder drives memorization, and shows augmentations/embedding noise can reduce memorization while improving ImageNet linear probing.
Problem
Memorization in multi-modal contrastive models like CLIP is poorly understood; existing definitions (supervised label-based, SSLMem for single encoders, Déjà Vu) do not fit CLIP's joint image-text representations.
Approach
Define CLIPMem as the difference in joint image-text alignment scores (cosine similarity minus similarity to random test samples in both directions) between a CLIP model trained with a pair and a reference model trained without it, in a leave-one-out style; then use it to diagnose memorized samples and mitigate memorization via caption augmentation, multiple captions, and embedding noise.
Key results
- Mis-captioned samples in a poisoned COCO CLIP model reach CLIPMem 0.586 versus 0.440 for clean candidates.
- Text encoder SSLMem on candidates averages 0.209 versus 0.168 for the image encoder, and CLIPMem reaches 0.91 on candidates versus much weaker SSLMem signals.
- Adding Gaussian noise N(0.15) to text embeddings lowers CLIPMem from 0.438 to 0.417 while raising ImageNet linear probing from 63.11% to 65.34%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size from ViT-Base to ViT-Large | CLIP · COCO | CLIPMem: 0.438 -> 0.457 | Hurt |
| Increase model size from ViT-Base to ViT-Large | CLIP · ImageNet | linear probing accuracy: 63.11% $\pm$ 0.91% -> 67.04\% $\pm$ 1.05\% | Helped |
| Use five COCO captions equally often instead of one caption (ablation) | CLIP ViT-Base · COCO | CLIPMem: 0.438 -> 0.423 | Helped |
| Use five COCO captions equally often instead of one caption (ablation) | CLIP ViT-Base · ImageNet | linear probing accuracy: 63.11\% $\pm$ 0.91\% -> 64.88\% $\pm$ 0.83\% | Helped |
| Randomly select among five captions rather than balance their use (ablation) | CLIP ViT-Base · ImageNet | linear probing accuracy: 63.11\% $\pm$ 0.91\% -> 62.44\% $\pm$ 1.18\% | Hurt |
| Use five GPT-3.5 captions equally during training instead of one COCO caption (ablation) | CLIP ViT-Base · COCO | CLIPMem: 0.438 -> 0.411 | Helped |
| Use five GPT-3.5 captions equally during training instead of one COCO caption (ablation) | CLIP ViT-Base · ImageNet | linear probing accuracy: 63.11\% $\pm$ 0.91\% -> 64.47 $\pm$ 0.72\% | Helped |
| Add Gaussian noise N(0.15) to text embeddings (ablation) | CLIP ViT-Base · COCO | CLIPMem: 0.438 -> 0.417 | Helped |
| Add Gaussian noise N(0.15) to text embeddings (ablation) | CLIP ViT-Base · ImageNet | linear probing accuracy: 63.11\% $\pm$ 0.91\% -> 65.34\% \ \pm \ 0.84\% | Helped |
| Increase text-embedding noise to N(0.50) (ablation) | CLIP ViT-Base · ImageNet | linear probing accuracy: 65.34\% \ \pm \ 0.84\% -> 57.04\% $\pm$ 1.11\% | Hurt |
| Train once on non-repeated YFCC100M data rather than 100 epochs on COCO (ablation) | CLIP ViT-Base · ImageNet | linear probing accuracy: 63.11\% $\pm$ 0.91\% -> 64.83\% $\pm$ 1.04\% | Helped |
Surprising findings
- Removing the most memorized samples can improve CLIP ImageNet linear probing up to a cutoff of roughly 400-800 (COCO) or 1600-3200 (CC3M) removed samples, unlike supervised and SSL where removal hurts.
- Unlike the trade-off reported for supervised and SSL learning, reducing memorization via augmentation and noise improved utility in CLIP.
Limitations
- Experiments are mainly on COCO with small models (ViT-Base, ViT-Large) and a 75K-sample CC3M subset rather than full-scale pretraining.
- Memorization estimation requires training pairs of models, and the approximation uses subsets of candidates rather than per-sample retraining.
Experimental setup
Datasets: COCO CC3M YFCC100M ImageNet CIFAR10
Benchmarks: ImageNet linear probing
Metrics: CLIPMem linear probing accuracy SSLMem UnitMem
Baselines: SSLMem (image encoder) SSLMem (text encoder) Déjà Vu text randomization CLIP similarity filtering random sample removal
Builds on: SSLMem Déjà Vu memorization LaCLIP UnitMem
Closest papers by meaning
- Déjà Vu Memorization in Vision–Language Models (NeurIPS 2024 · similarity 0.89)
- Targeted Forgetting of Image Subgroups in CLIP Models (CVPR 2025 · similarity 0.88)
- CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation (CVPR 2025 · similarity 0.88)
- Exploring Local Memorization in Diffusion Models via Bright Ending Attention (ICLR 2025 · similarity 0.88)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.87)
- Detecting, Explaining, and Mitigating Memorization in Diffusion Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.