Research map / Label-efficient and robust learning / Self-supervised and contrastive
Memorization in Self-Supervised Learning Improves Downstream Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSLMem, a training-method-agnostic per-point memorization score for self-supervised encoders based on the alignment difference between encoders trained with and without a data point, and shows that memorization correlates with downstream generalization.
Problem
Memorization in self-supervised learning has not been defined or studied in a general way, because existing supervised definitions rely on labels and existing SSL memorization work (Déjà Vu) requires labeled data and cropping augmentations.
Approach
Define a leave-one-out memorization score for SSL encoders as the difference in expected augmentation alignment (l2 distance between representations of augmented views) between encoders trained without and with a data point. The score operates directly on encoder representations, so it is independent of SSL objective, augmentation, downstream task, and labels.
Key results
- SSLMem average memorization for MAE ViT-tiny on CIFAR10 is 0.307 vs. candidate mean 0.307 and independent-set mean -0.312; removing the 16k most memorized CIFAR10 points hurts linear-probing accuracy more than removing random points.
- On ImageNet, removing 20k most memorized points lowers ADE20K mIoU to 43.8 vs. 44.4 for random removal.
- Stronger alignment regularization lowers memorization but also lowers downstream linear probing accuracy, suggesting memorization is needed for generalization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the most memorized pretraining samples instead of the same number of random samples (ablation) | MAE ViT-tiny pretrained on CIFAR10 · CIFAR10, CIFAR100, and STL10 linear probing | linear probing accuracy | Hurt |
| remove memorized samples instead of random samples (ablation) | MAE ViT-base pretrained on ImageNet · ADE20K | mIoU: 45.4 -> 44.8; 45.4 -> 43.8 | Hurt |
| remove memorized samples instead of random samples (ablation) | MAE ViT-base pretrained on ImageNet · NYU-Depth v2 | RMSE: 0.289 -> 0.295; 0.289 -> 0.311 | Hurt |
| increase alignment-penalty regularization strength λ (ablation) | MAE ViT-tiny · CIFAR10 and SVHN | linear probing accuracy | Hurt |
| apply differential privacy during training (ablation) | MAE ViT-tiny · CIFAR10 | linear probing accuracy: 69.40% $\pm$ 1.12% -> 54.22% $\pm$ 0.98%; 69.40% $\pm$ 1.12% -> 33.66% $\pm$ 1.76% | Hurt |
| use 75% rather than 50%, 80%, or 90% masking (ablation) | MAE ViT-tiny · CIFAR10 | linear probing accuracy: 62.09\% $\pm$ 0.43\% -> 67.40\% $\pm$ 1.10\%; 65.06\% $\pm$ 1.35\% -> 67.40\% $\pm$ 1.10\%; 58.77\% $\pm$ 1.26\% -> 67.4 | Mixed |
| use cropping rather than noise addition or masking (ablation) | SimCLR ResNet50 · CIFAR10 | average SSLMem memorization: 0.319 $\pm$ 0.006 -> 0.322 $\pm$ 0.010; 0.288 $\pm$ 0.012 -> 0.322 $\pm$ 0.010 | Helped |
| measure memorization with alternative augmentations instead of SimCLR's original augmentations (ablation) | SimCLR ResNet50 · CIFAR10 | average SSLMem memorization: 0.339 ± 0.011; 0.321 ± 0.014; 0.308 ± 0.009; 0.328 ± 0.011; 0.298 ± 0.006 | No clear effect |
| increase overlap between f and g training sets from 80% to 85% (ablation) | SimCLR ResNet50 · CIFAR10 | average SSLMem memorization: 0.339 $\pm$ 0.011 -> 0.337 $\pm$ 0.010 | No clear effect |
| decrease overlap between f and g training sets from 80% to 70% (ablation) | SimCLR ResNet50 · CIFAR10 | average SSLMem memorization: 0.339 $\pm$ 0.011 -> 0.325 $\pm$ 0.008 | Hurt |
| increase SSL training duration (ablation) | DINO ResNet50 · CIFAR10 | average SSLMem memorization: 0.314 $\pm$ 0.011 -> 0.327 $\pm$ 0.009 | Helped |
| train DINO for 300 rather than 100 epochs (ablation) | ResNet50 · CIFAR10 | linear probing accuracy: 69.12% $\pm$ 0.87% -> 75.39% $\pm$ 1.15% | Helped |
Surprising findings
- MAE shows the lowest SSLMem scores but the highest Déjà Vu memorization, a divergence between the two metrics.
- Supervised ResNet50 trained 10 epochs matches SSL 300-epoch memorization despite different memorized points, with only about 50% overlap among top memorized samples.
Limitations
- Exact memorization score requires training many encoders; the paper approximates it with a 80%/10%/10% split and a single pair of encoders.
- Differential privacy reduces memorization but also strongly reduces downstream accuracy, so the trade-off is not resolved.
- Results are reported for relatively small ViT-tiny and ResNet50 setups on small image datasets.
Experimental setup
Datasets: CIFAR10 CIFAR100 SVHN STL10 ImageNet ADE20K NYU-Depth v2
Benchmarks: CIFAR10 CIFAR100 SVHN STL10 ImageNet ADE20K NYU-Depth v2
Metrics: SSLMem memorization score linear probing accuracy mIoU RMSE Kendall's tau Deja Vu memorization fraction
Baselines: Déjà Vu memorization EncoderMI SimCLR DINO VICReg MAE
Builds on: SimCLR MAE DINO SimSiam Déjà Vu memorization EncoderMI DP-SGD
Closest papers by meaning
- Localizing Memorization in SSL Vision Encoders (NeurIPS 2024 · similarity 0.91)
- The Pitfalls of Memorization: When Memorization Hurts Generalization (ICLR 2025 · similarity 0.88)
- Towards Memorization Estimation: Fast, Formal and Free (ICML 2025 · similarity 0.88)
- Captured by Captions: On Memorization and its Mitigation in CLIP Models (ICLR 2025 · similarity 0.87)
- Rethinking LLM Memorization through the Lens of Adversarial Compression (NeurIPS 2024 · similarity 0.86)
- Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.