Research map / Label-efficient and robust learning / Self-supervised and contrastive
Learning from Memory: Non-Parametric Memory Augmented Self-Supervised Learning of Visual Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaSSL, a negative-free SSL method that matches view-to-memory similarity distributions over randomly sampled memory blocks from a FIFO non-parametric memory, avoiding prototypes and extra regularizers.
Problem
Self-supervised clustering/join-embedding methods are unstable and prone to collapse, and prototype-based heads add many trainable parameters and compute cost, especially when ViTs use patch tokens.
Approach
Replace learnable prototypes with a non-parametric FIFO memory of past student/teacher [CLS] representations; compare each view's cosine similarity distribution over randomly sampled, disjoint memory blocks and enforce cross-view consistency with cross-entropy, without extra anti-collapse regularizers.
Key results
- ViT-B/16 k-NN transfer at k=20 beats DINO/iBOT on 5 of 8 datasets, with gains of +2.6 on Aircraft and +4.6 on GTSRB.
- ViT-B/16 ImageNet-1M linear probing reaches 79.6 vs iBOT 79.5 at 400 epochs; ViT-S/16 800-epoch training is about 2% faster in time and uses 15.1G vs 15.4G (DINO) memory.
- ViT-B/16 image retrieval outperforms DINO on 3 of 4 Oxford/Paris settings; Copydays ViT-B mAP 87.6 vs DINO 86.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaSSL instead of DINO | ViT-B/16 representations · Transfer datasets, k=20 average | top-1 accuracy: 62.00625229 -> 63.36438755 | Helped |
| Use MaSSL instead of DINO | ViT-B/16 representations · GTSRB, k=20 | top-1 accuracy: +4.6 | Helped |
| Use MaSSL instead of iBOT | ViT-B/16 representations · Oxford-5k Hard retrieval | mAP: 14.25 -> 14.12 | Hurt |
| Use MaSSL instead of DINO and iBOT | ViT-B/16 representations · ImageNet-1M low-shot, 1% labels, k-NN | accuracy: 66.298 -> 68.804 | Helped |
| Use MaSSL instead of iBOT | ViT-B/16 representations · ImageNet-9 background variants | accuracy | Mixed |
| Increase memory size from K=8192 to K=65536 (ablation) | MaSSL · ImageNet-1M | k-NN top-1 accuracy: 67.824 -> 71.862 | Helped |
| Increase memory size from K=65536 to K=131072 (ablation) | MaSSL · ImageNet-1M | k-NN top-1 accuracy: 71.862 -> 71.924 | No clear effect |
| Increase memory block size from Nb=512 to Nb=16384 (ablation) | MaSSL · ImageNet-1M | k-NN top-1 accuracy: 67.804 -> 71.862 | Helped |
| Increase memory block size from Nb=16384 to Nb=32768 (ablation) | MaSSL · ImageNet-1M | k-NN top-1 accuracy: 71.862 -> 70.644 | Hurt |
| Use stochastic rather than blockwise memory-block sampling (ablation) | MaSSL · ImageNet-1M, Nb=512 | k-NN top-1 accuracy: 0.1 -> 67.8 | Helped |
| Use stochastic rather than blockwise memory-block sampling (ablation) | MaSSL · ImageNet-1M, Nb=1024 | k-NN top-1 accuracy: 0.1 -> 68.5 | Helped |
| Use stochastic rather than blockwise memory-block sampling (ablation) | MaSSL · ImageNet-1M, Nb=2048 | k-NN top-1 accuracy: 0.1 -> 70.0 | Helped |
Surprising findings
- Blockwise (contiguous) memory block sampling collapses regardless of block size (0.1% k-NN), while stochastic sampling works.
- The method trains only on the [CLS] token yet beats iBOT (which uses patch tokens) on 4 of 7 ImageNet-9 background-robustness variants.
Limitations
- Reported comparisons often rely on reproduced or adapted baselines; some low-shot baselines were re-run due to reproducibility issues.
- Gains over DINO/iBOT are mostly small (often under 1 point) and not uniform across tasks, e.g. linear probing on par with iBOT.
Experimental setup
Datasets: ImageNet-1M ImageNet-LT ImageNet-9 Oxford-IIIT Pet Oxford Flowers-102 FGVC Aircraft Stanford Cars Country-211 Food-101 STL-10 GTSRB CIFAR-10
Benchmarks: ImageNet-1M linear probing ImageNet-1M k-NN ImageNet 1%/10% low-shot ImageNet-LT ImageNet-9 robustness Oxford-5k Paris-6k INRIA Copydays ImageNet-1% clustering CIFAR-10 clustering transfer learning k-NN and logistic regression
Metrics: top-1 accuracy k-NN accuracy mAP NMI AMI ARI training time GPU memory
Baselines: DINO iBOT MoCo-v3 MAE NNCLR CARP DeiT (Sup)
Builds on: DINO PAWS iBOT MoCo
Closest papers by meaning
- Revisiting Self-Supervised Visual Representation Learning (CVPR 2019 · similarity 0.88)
- Towards flexible perception with visual memory (ICML 2025 · similarity 0.87)
- RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning (NeurIPS 2022 · similarity 0.87)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.86)
- Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.86)
- Self-Organizing Visual Prototypes for Non-Parametric Representation Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.