Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Organizing Visual Prototypes (SOP), a non-parametric SSL method where each prototype is a set of k nearest-neighbor support embeddings from a memory bank, with a new patch-level SOP-MIM loss, evaluated on ImageNet linear/k-NN, detection, segmentation, retrieval and robustness.
Problem
Prototypical self-supervised learning relies on single (learnable or non-parametric) prototypes that may underrepresent their region in feature space and requires over-clustering with many prototypes, which is costly and suboptimal.
Approach
Replace each prototype with a Self-Organizing Prototype (SOP): an anchor sampled from a memory bank plus its k nearest-neighbor support embeddings, whose soft-weighted votes (similarity-based contributions) produce view-to-region likelihoods; also a non-parametric masked image modeling loss that uses patch-level SOPs instead of learned tokenizers.
Key results
- With ViT-L/16, SOP reaches 79.2% k-NN top-1 on ImageNet, +1.2 over iBOT.
- With ViT-B/16 and 1024 SOPs, SOP gets 69.5% k-NN top-1 vs iBOT 67.6% in ablation; ViT-B/16 mAP on Revisited Oxford Hard improves to 17.52 vs iBOT 14.25.
- On COCO with ViT-B, SOP gives 51.4 AP^b vs iBOT 51.2 and 38.7 ADE20K mIoU (linear) vs iBOT 38.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SOP instead of iBOT | ViT-L/16 representations · ImageNet-1M k-NN evaluation | top-1 accuracy: +1.2% | Helped |
| Use SOP instead of iBOT | ViT-B representations · COCO object detection | APb: +0.2 | Helped |
| Use SOP instead of iBOT | ViT-B representations · COCO instance segmentation | APm: +0.1 | Helped |
| Use SOP instead of iBOT | ViT-B representations · ADE20K linear probing | mIoU: +0.4 | Helped |
| Use SOP instead of iBOT | ViT-B representations · ADE20K fine-tuning | mIoU: +0.6 | Helped |
| Use SOP instead of iBOT | ViT-B/16 representations · DAVIS 2017 video segmentation | mean contour accuracy Fm: +1.3 | Helped |
| Add SOP-MIM and [CLS] losses together (ablation) | SOP pretraining · ImageNet-1M | k-NN top-1 accuracy: 0.9% | Helped |
| Add support embeddings to the [CLS] global loss (ablation) | SOP pretraining · ImageNet-1M | k-NN top-1 accuracy | Helped |
| Add multiple support embeddings to SOP-MIM (ablation) | SOP pretraining · ImageNet-1M | k-NN top-1 accuracy | No clear effect |
| Replace soft SE contribution weights with one-hot equal weights (ablation) | SOP pretraining · ImageNet-1M | k-NN top-1 accuracy: 70.0 vs 69.724 | Hurt |
| Use average patch embeddings instead of the [CLS] token (ablation) | SOP pretraining · ImageNet-1M | k-NN top-1 accuracy: 70.0 vs 67.8 | Hurt |
| Increase the number of independent [CLS] and [MIM] tasks per iteration (ablation) | SOP pretraining · ImageNet-1M | k-NN top-1 accuracy: 68.0; 68.5; 68.6; 69.6; 70.0 | Helped |
Surprising findings
- Using a single SE (the anchor itself) for the SOP-MIM loss performs as well as multiple SEs, unlike the global loss which benefits from up to 8 SEs.
- Keeping anchors fixed during pre-training leads to training collapse, indicating anchor randomization acts as a regularizer.
Limitations
- Gains over iBOT are often modest (e.g., +0.2 AP^b on COCO, +0.1 AP^m), and ViT-S performance is similar to baselines.
- Fine-tuning on small datasets shows saturation where most methods perform similarly after 1000 epochs.
- Authors note hyperparameter tuning, extra regularizers and scaling remain unexplored.
Experimental setup
Datasets: ImageNet-1M CIFAR-10 CIFAR-100 iNaturalist 2018 iNaturalist 2019 Oxford Flowers Stanford Cars COCO ADE20K ImageNet-9 Oxford and Paris revisited DAVIS 2017
Benchmarks: ImageNet-1M k-NN ImageNet-1M linear probing ImageNet 1%/10%/100% fine-tuning COCO ADE20K CIFAR-10 CIFAR-100 iNaturalist 2018 iNaturalist 2019 Oxford Flowers Stanford Cars Revisited Oxford and Paris
Metrics: top-1 accuracy AP^b AP^m mIoU mAP J&F robustness accuracy
Baselines: DINO iBOT MaSSL EsViT BEiT MoCo-v3 I-JEPA MAE
Builds on: DINO iBOT MaSSL
Closest papers by meaning
- From Prototypes to General Distributions: An Efficient Curriculum for Masked Image Modeling (CVPR 2025 · similarity 0.88)
- Revisiting Self-Supervised Visual Representation Learning (CVPR 2019 · similarity 0.88)
- Self-Supervised Visual Representation Learning from Hierarchical Grouping (NeurIPS 2020 · similarity 0.87)
- Unsupervised Feature Learning via Non-Parametric Instance Discrimination (CVPR 2018 · similarity 0.87)
- Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.86)
- Learning from Memory: Non-Parametric Memory Augmented Self-Supervised Learning of Visual Features (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.