Research map / Vision-language and multimodal / CLIP and zero-shot
Déjà Vu Memorization in Vision–Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VL-Déjà-Vu, a two-model kNN test that measures how many ground-truth objects in a training image can be recovered from its caption beyond correlation, and finds OpenCLIP memorizes at 1M–50M scale, with text masking as the most effective mitigation.
Problem
Whether vision-language models trained with contrastive learning memorize training data beyond what generalization or dataset-level correlations explain, which is hard to measure since VLMs have no decoder and captions omit many objects.
Approach
Adapts the foreground-background Déjà Vu test to VLMs: encode a training caption, retrieve k nearest public images with a target and a reference model trained on disjoint sets, detect objects with Detic, and measure the precision/recall gap between the two models at sample and population levels.
Key results
- Text masking with ratio 0.5 reduces PPG from 9.1% to 3.0%, PRG from 9.2% to 1.9%, and AUCG from 0.034 to 0.007 on ImageNet public set, with ImageNet zero-shot accuracy dropping from 25.2% to 20.3%.
- Masking ratio 0.3 reduces ImageNet zero-shot accuracy only from 25.2% to 24.1% while cutting PPG to 3.4%.
- Memorization gap decreases as training size grows from 1M to 50M on filtered LAION, but remains significant at 10M and 50M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| set temperature to 25 instead of 100 (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet public set | PPG: 0.091 -> 0.135 | Hurt |
| set temperature to 25 instead of 100 (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet zero-shot | accuracy: 0.252 -> 0.217 | Hurt |
| set temperature to 200 instead of 100 (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet public set | PPG: 0.091 -> 0.087 | No clear effect |
| set weight decay to 0.03 instead of 0.1 (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet public set | PPG: 0.091 -> 0.092 | Hurt |
| set weight decay to 0.3 instead of 0.1 (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet zero-shot | accuracy: 0.252 -> 0.303 | Helped |
| set weight decay to 0.3 instead of 0.1 (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet public set | PPG: 0.091 -> 0.084 | Helped |
| randomly mask 30% of text tokens during training (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet public set | PPG: 0.091 -> 0.025 | Helped |
| randomly mask 30% of text tokens during training (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet zero-shot | accuracy: 0.252 -> 0.241 | Hurt |
| randomly mask 50% of text tokens during training (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet public set | AUCG: 0.033 -> 0.007 | Helped |
| randomly mask 50% of text tokens during training (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet zero-shot | accuracy: 0.252 -> 0.203 | Hurt |
| stop training at 20 epochs instead of 200 (ablation) | OpenCLIP ViT-B-32 trained on 10M filtered LAION · ImageNet public set | memorization | Helped |
| train for more epochs (ablation) | OpenCLIP ViT-B-32 · ImageNet | memorization | Hurt |
Surprising findings
- Text masking also improves ARO compositional benchmark accuracy, suggesting less bag-of-words behavior.
- Default temperature of 100 was already near-optimal for the utility-memorization trade-off.
Limitations
- Requires access to two models (target and reference) and the underlying training set, so it cannot be applied to off-the-shelf pretrained models.
- Object annotations from Detic are imperfect, though the relative gap is argued to be robust.
- Uses Shutterstock, a private dataset, limiting reproducibility.
Experimental setup
Datasets: filtered LAION MS-COCO Shutterstock ImageNet
Benchmarks: ImageNet zero-shot accuracy ARO
Metrics: precision gap recall gap F-score gap AUC gap ImageNet zero-shot accuracy
Baselines: CLIP (pretrained, 400M private data)
Builds on: Déjà Vu memorization (image representation models) CLIP OpenCLIP Detic
Closest papers by meaning
- Measuring Dejavu Memorization Efficiently (NeurIPS 2024 · similarity 0.92)
- Captured by Captions: On Memorization and its Mitigation in CLIP Models (ICLR 2025 · similarity 0.89)
- Quantifying Cross-Modality Memorization in Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models (CVPR 2026 · similarity 0.88)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.