Research map / Label-efficient and robust learning / Self-supervised and contrastive
Clustering Properties of Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReSA, a positive-feedback SSL method that uses Sinkhorn-Knopp self-clustering on encoder outputs (encodings) to build soft assignment targets for a cross-entropy alignment loss between views.
Problem
Self-supervised joint-embedding methods learn strongly clustered representations, but these clustering properties are rarely exploited to improve SSL training itself.
Approach
Measure that encoder outputs (encodings) cluster better and more stably than projector embeddings, then compute online Sinkhorn-Knopp self-similarity assignments over the batch's encodings (no prototypes, no gradient through assignment) and use them as soft targets in a cross-entropy loss on embeddings across views.
Key results
- ReSA reaches 93.02% k-NN on CIFAR-10 and 66.83% k-NN on CIFAR-100, versus 90.03% and 61.90% for the next best method INTL.
- With ResNet-50 on ImageNet, ReSA achieves 71.9% linear accuracy after 100 epochs (batch 256) and 75.2% after 800 epochs, the latter matching the multi-crop SwAV/DINO level.
- ReSA trains faster than DINO and MoCoV3 on ViT-S/16 (0.12 vs 0.21-0.22 h/epoch), but memory is similar to MoCoV3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a momentum network (ablation) | ReSA · ImageNet, batch size 256 | linear accuracy: 69.7 -> 71.9 | Helped |
| add a predictor (ablation) | ReSA · ImageNet, batch size 256 | linear accuracy: 70.8 -> 71.9 | Helped |
| set temperature to 0.4 rather than the tested alternatives (ablation) | ReSA · ImageNet, batch sizes 256 and 1024 | linear accuracy: 71.2, 71.4, 71.9, 71.7; 70.9, 71.1, 71.3, 71.2 | Helped |
| derive self-assignments from embeddings instead of encodings (ablation) | ReSA · ImageNet-100 | linear accuracy | Hurt |
| use weak augmentation instead of standard augmentation (ablation) | SwAV · CIFAR-10 and CIFAR-100 | linear accuracy: 89.17 -> 88.79; 64.88 -> 64.21 | Hurt |
| use weak augmentation instead of standard augmentation (ablation) | VICReg · CIFAR-10 and CIFAR-100 | linear accuracy: 92.07 -> 91.35; 68.54 -> 67.27 | Hurt |
| use weak augmentation instead of standard augmentation (ablation) | MoCoV3 · CIFAR-10 and CIFAR-100 | linear accuracy: 93.10 -> 93.05; 68.83 -> 68.72 | Hurt |
| use weak augmentation instead of standard augmentation (ablation) | MoCoV3 · CIFAR-10 | k-nn accuracy: 89.06 -> 89.09 | Mixed |
| remove ColorJitter, Grayscale, or GaussianBlur from ReSA's clustering-view augmentation (ablation) | ReSA · ImageNet-100 | clustering performance | Helped |
| remove HorizontalFlip from ReSA's clustering-view augmentation (ablation) | ReSA · ImageNet-100 | validation accuracy | Hurt |
| replace ResizedCrop with fixed CenterCrop (ablation) | ReSA · ImageNet-100 | representation quality | Hurt |
| use weak augmentation with only ResizedCrop and HorizontalFlip (ablation) | ReSA · ImageNet-100 | representation quality | Helped |
Surprising findings
- Projector embeddings (not encodings) are used for the loss, yet the loss on embeddings with encoding-derived assignments reportedly struggles to converge when assignments come from embeddings instead.
- Weak augmentation (ResizedCrop and HorizontalFlip only) helps ReSA but does not help SwAV, VICReg or MoCoV3.
Limitations
- Gains on long-tailed CIFAR100-LT are much lower and early training suffers from noisy clustering; the paper notes early clustering errors are a concern.
- ViT results are preliminary with no extensive training tricks, and the ViT setup was not tuned for ReSA.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet COCO 2017 CUB-200-2011 Oxford-IIIT Pets Food-101 Oxford Flowers-102 CIFAR100-LT
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet-1K linear probe COCO detection and instance segmentation CUB-200-2011 Pets-37 Food-101 Flowers-102 CIFAR-100 coarse-grained
Metrics: top-1 accuracy (linear) k-NN accuracy AP silhouette coefficient adjusted rand index memory training time
Baselines: SimCLR BYOL SwAV SimSiam MoCoV3 W-MSE DINO Barlow Twins VICReg CW-RGP INTL MEC
Builds on: SwAV InfoNCE solo-learn MoCoV3 INTL
Closest papers by meaning
- Every Node Is Different: Dynamically Fusing Self-Supervised Tasks for Attributed Graph Clustering (AAAI 2024 · similarity 0.87)
- RPSC: Robust Pseudo-Labeling for Semantic Clustering (AAAI 2024 · similarity 0.87)
- On the Importance of Embedding Norms in Self-Supervised Learning (ICML 2025 · similarity 0.86)
- Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining (ICML 2024 · similarity 0.86)
- Exploring Structural Degradation in Dense Representations for Self-supervised Learning (NeurIPS 2025 · similarity 0.86)
- Revisiting Self-Supervised Heterogeneous Graph Learning from Spectral Clustering Perspective (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.