Research map / Vision-language and multimodal / Human and embodied modeling
Measuring Dejavu Memorization Efficiently
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes one-model déjà vu memorization tests that replace the second reference model with a ResNet50 or Naive Bayes classifier (image) or a GTE text-embedding nearest-neighbor reference (VLM), enabling memorization measurement of pre-trained open-source models without retraining.
Problem
Measuring déjà vu memorization in representation learning models normally requires training a second reference model on disjoint training splits, which is infeasible for large pre-trained open-source image and vision-language models.
Approach
Estimate dataset-level correlations with cheap reference predictors (a ResNet50 background-crop classifier, a Naive Bayes classifier over detected crop objects, or a pre-trained text embedding model with nearest-neighbor caption matching) so that memorization of an off-the-shelf model can be measured with a single model.
Key results
- On ImageNet, the one-model ResNet50 and Naive Bayes tests identify similar déjà vu memorization to the two-model KNN test for VICReg and Barlow Twins, with DINO showing substantially lower memorization.
- For a CLIP model trained on 40M Shutterstock pairs, the one-model GTE-based test gives 0.07 PPG and 0.06 PRG for top-10 objects with 10 NNs, versus 0.06 PPG and 0.06 PRG for the two-model test.
- Pre-trained open-source models (e.g., YFCC15M CLIP at 0.07 PPG and PRG) show lower aggregate memorization than the same models trained on subsets of the data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the separately trained reference VLM with a GTE text-embedding reference | CLIP déjà vu memorization measurement on Shutterstock · Shutterstock, top-10 objects with 10 nearest neighbors | PPG: 0.06 -> 0.07 | Mixed |
| Replace the separately trained reference VLM with a GTE text-embedding reference | CLIP déjà vu memorization measurement on Shutterstock · Shutterstock, top-10 objects with 10 nearest neighbors | PRG: 0.06 -> 0.06 | No clear effect |
| Increase the number of crop-annotation features in the Naive Bayes reference (ablation) | ImageNet image representation memorization measurement · ImageNet | déjà vu score | Hurt |
| Use open-source pretrained models rather than the same models trained on a smaller subset | VICReg, Barlow Twins, and DINO · ImageNet | déjà vu memorization | Hurt |
| Use VICReg or Barlow Twins rather than DINO | Open-source image representation models · ImageNet | déjà vu memorization | Mixed |
Surprising findings
- Pairwise sample-level agreement among the reference models is only about 40%, even though the top-5 most correlated classes are identical across methods.
- The one-model test slightly overestimates memorization relative to the two-model test in the VLM setting.
Limitations
- Alternative dataset-level correlation estimators may be poor approximations to the Bayes-optimal predictor or may themselves memorize training data.
- Additional object annotations (Grounded-SAM, Detic) may be lower quality and bias results.
- Reference models disagree substantially at the sample level (about 40% agreement), so one-model tests can overestimate memorization.
Experimental setup
Datasets: ImageNet Shutterstock (239M image-caption pairs, private) YFCC15M
Benchmarks: ImageNet Shutterstock public set
Metrics: déjà vu score déjà vu score at p% population precision gap (PPG) population recall gap (PRG) AUC gap
Baselines: two-model déjà vu test KNN classifier VICReg Barlow Twins DINO
Builds on: déjà vu memorization CLIP OpenCLIP
Closest papers by meaning
- Déjà Vu Memorization in Vision–Language Models (NeurIPS 2024 · similarity 0.92)
- Towards Memorization Estimation: Fast, Formal and Free (ICML 2025 · similarity 0.86)
- Memorization Through the Lens of Curvature of Loss Function Around Samples (ICML 2024 · similarity 0.85)
- On the Memorization of Consistency Distillation for Diffusion Models (NeurIPS 2026 · similarity 0.84)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.84)
- A Geometric Framework for Understanding Memorization in Generative Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.