Research map / Architectures and efficiency / Pruning and efficient tuning
Localizing Memorization in SSL Vision Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LayerMem and UnitMem, label-free forward-pass metrics that localize data-point memorization in SSL vision encoders per layer and per unit, and compares the patterns with supervised learning.
Problem
Prior work shows SSL vision encoders memorize individual training points, but little is known about where in the encoder this memorization happens; existing localization methods require labels or are confined to supervised learning or language models.
Approach
Normalized SSLMem-based per-layer score (LayerMem) and a label-free unit-level score (UnitMem) that compares a unit's maximum mean activation on one training point against the mean over the other training points, averaged over augmentations, computed in a forward pass.
Key results
- LayerMem increases with depth in a ResNet9 SimCLR encoder on CIFAR10 (0.091 at layer 1 to 0.308 at layer 8), but not monotonically.
- Highly memorizing units are spread across the entire encoder, including the first layer, and most memorization in ViTs occurs in fully-connected layers rather than attention layers.
- Pruning the most memorized units per layer drops CIFAR10 linear probing accuracy from 70.44% to 53.04% with 10% of units removed, versus 58.09% for random pruning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the most memorizing layer(s) rather than only the last layer(s) (ablation) | ResNet9 SimCLR · STL10 | accuracy (%): 53.0% ± 0.86% vs 52.7% ± 0.97%; 56.7% ± 0.84% vs 55.3% ± 0.77%; 57.9% ± 0.79% vs 56.5% ± 0.95% | Helped |
| Prune the top UnitMem units per layer instead of random units (ablation) | ResNet9 SimCLR · CIFAR10 linear probing | accuracy (%): 58.09 ± 1.76 -> 53.04 | Hurt |
| Prune the low UnitMem units per layer instead of random units (ablation) | ResNet9 SimCLR · CIFAR10, SVHN, STL10 linear probing | accuracy (%): CIFAR10: 58.09 ± 1.76 -> 62.58; SVHN: 67.04 ± 2.44 -> 72.26; STL10: 55.71 ± 2.18 -> 59.26 | Mixed |
| Prune the top UnitMem units rather than random units (ablation) | ResNet9 SimCLR · SVHN linear probing | accuracy (%): 67.04 ± 2.44 -> 63.84 | Hurt |
| Prune the top UnitMem units rather than random units (ablation) | ResNet9 SimCLR · STL10 linear probing | accuracy (%): 55.71 ± 2.18 -> 50.94 | Hurt |
| Increase the fraction of pruned top UnitMem units per layer from 10% to 20% (ablation) | ResNet9 SimCLR · CIFAR10, SVHN, STL10 linear probing | accuracy (%): CIFAR10: 53.04 -> 48.30; SVHN: 63.84 -> 55.88; STL10: 50.94 -> 43.18 | Hurt |
| Measure UnitMem with augmentations rather than without augmentations (ablation) | ResNet9 SimCLR and supervised ResNet9 · CIFAR100 | UnitMem | Helped |
| Vary the number or composition of data points used to measure UnitMem (ablation) | ResNet9 SimCLR · CIFAR100 | UnitMem | No clear effect |
| Use the median rather than the mean for UnitMem (ablation) | ResNet9 SimCLR · CIFAR10 | UnitMem | No clear effect |
| Use stronger training augmentations (ablation) | ResNet9 SimCLR · CIFAR10 | UnitMem | Hurt |
| Increase training weight decay (ablation) | ResNet9 SimCLR · CIFAR10 | UnitMem | Hurt |
| Increase differential privacy (ablation) | ViT-Tiny MAE · CIFAR10 | average UnitMem | Hurt |
Surprising findings
- SSL encoders memorize individual data points more than whole classes throughout all layers, whereas supervised models increasingly memorize classes in deeper layers.
- Measuring UnitMem without augmentations under-reports memorization.
Limitations
- Results are reported for small-scale vision encoders and datasets (CIFAR10, SVHN, STL10, ImageNet subsets) rather than large foundation models.
- Fine-tuning and pruning applications are only preliminary demonstrations.
Experimental setup
Datasets: ImageNet ILSVRC-2012 CIFAR10 CIFAR100 SVHN STL10
Benchmarks: CIFAR10 SVHN STL10 ImageNet
Metrics: LayerMem UnitMem linear probing accuracy SSLMem Kendall's tau
Baselines: SimCLR DINO MAE SimSiam supervised learning model (cross-entropy)
Builds on: SSLMem class selectivity SimCLR DINO MAE
Closest papers by meaning
- Memorization in Self-Supervised Learning Improves Downstream Generalization (ICLR 2024 · similarity 0.91)
- Exploring Local Memorization in Diffusion Models via Bright Ending Attention (ICLR 2025 · similarity 0.86)
- Captured by Captions: On Memorization and its Mitigation in CLIP Models (ICLR 2025 · similarity 0.86)
- Probing the Mid-level Vision Capabilities of Self-Supervised Learning (CVPR 2025 · similarity 0.86)
- Towards Memorization Estimation: Fast, Formal and Free (ICML 2025 · similarity 0.86)
- Déjà Vu Memorization in Vision–Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.