Research map / Label-efficient and robust learning / Self-supervised and contrastive
Collapse-Proof Non-Contrastive Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CPLearn, a non-contrastive SSL projector and loss based on hyperdimensional computing (frozen random Rademacher codebook) that provably encourages decorrelated and clustered embeddings and avoids representation, dimensional, cluster and intracluster collapses.
Problem
Non-contrastive self-supervised learning suffers from failure modes (representation, dimensional, cluster and intracluster collapse) that existing heuristics (stop-gradient, momentum encoders, asymmetric heads) do not guarantee to avoid.
Approach
A projector producing normalized embeddings assigned softmax probabilities to a frozen quasi-orthogonal Rademacher dictionary, trained with an invariance (cross-entropy between augmented views) plus prior-matching loss; theory shows minima give hard, uniformly distributed, near-orthogonal assignments, so decorrelation and clustering arise without covariance computation or clustering layers.
Key results
- On CIFAR-10 with ResNet-8, CPLearn with 16384 codes reaches NMI 0.35 and linear accuracy 0.67 versus 0.29 NMI and 0.56 for SwAV-style baselines, outperforming all baselines on clustering.
- On ImageNet-100 with ViT-small, CPLearn with a large projector (131072 codes) reaches 70.9% NMI and 78.0% linear accuracy, while DINO reaches 64.3% NMI and 75.8% linear accuracy at its best settings.
- On CIFAR-10 with ResNet-18, CPLearn (16384) gets NMI 48.2 and linear accuracy 91.3, close to Barlow Twins' 92.2 linear accuracy but higher clustering NMI than all baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase dictionary size from c=10 or c=f to c=16384 (ablation) | CPLearn · SVHN, CIFAR-10, CIFAR-100 | NMI and accuracy: SVHN NMI: 0.11$\pm$0.01 -> 0.29$\pm$\textbf{0.00}; CIFAR-10 NMI: 0.28$\pm$0.02 -> \textbf{0.35}$\pm$\textbf{0.00}; CIFAR | Helped |
| Increase projector/dictionary size (ablation) | CPLearn · ImageNet-100 | NMI: 34.1\% -> 70.9\% | Helped |
| Increase projector/dictionary size (ablation) | CPLearn · ImageNet-100 | accuracy: 70.8\% -> 78.0\% | Mixed |
| Replace Tanh with L2-Norm activation (ablation) | CPLearn (c=16384) · SVHN | NMI: 0.29$\pm$0.00 -> \textbf{0.31}$\pm$\textbf{0.00} | Helped |
| Replace Tanh with L2-Norm activation (ablation) | CPLearn (c=16384) · CIFAR-100 | NMI: \textbf{0.59}$\pm$\textbf{0.00} -> 0.58$\pm$0.00 | Hurt |
| Replace Tanh with L2-Norm activation (ablation) | CPLearn (c=16384) · CIFAR-10 | NMI: \textbf{0.35}$\pm$\textbf{0.00} -> \textbf{0.35}$\pm$\textbf{0.00} | No clear effect |
| Replace Tanh with L2-Norm activation (ablation) | CPLearn (c=16384) · CIFAR-10 | accuracy: \textbf{0.68}$\pm$\textbf{0.00} -> \textbf{0.68}$\pm$\textbf{0.00} | No clear effect |
| Replace Tanh with L2-Norm activation (ablation) | CPLearn (c=16384) · CIFAR-100 | accuracy: 0.40$\pm$0.00 -> \textbf{0.41}$\pm$\textbf{0.00} | Helped |
| Increase projector size from 128 to 131072 (ablation) | CPLearn · ImageNet-100 | NMI: 34.1\% -> \textbf{70.9}\% | Helped |
| Increase projector size from 128 to 131072 (ablation) | CPLearn · ImageNet-100 | accuracy: 70.8\% -> \textbf{78.0}\% | Helped |
Surprising findings
- Barlow Twins remains the strongest linear-probe baseline on several settings, and CPLearn only matches rather than beats it in linear accuracy on CIFAR-10 with ResNet-18.
- Self-Classifier with an overcomplete dictionary (16384) degrades linear probing accuracy strongly (e.g. 64.9% on CIFAR-10 ResNet-18), suggesting large dictionaries alone do not help other methods.
Limitations
- Theory assumes an infinite-capacity backbone; finite-capacity guarantees are only sketched in an appendix discussion.
- Large dictionaries are costly to store and use as floating-point vectors; the authors note efficiency improvements via bipolar codes are future work.
- Experiments are limited to small-scale image benchmarks with ResNet-8/18 and ViT-small backbones and a fixed number of epochs.
Experimental setup
Datasets: SVHN CIFAR-10 CIFAR-100 ImageNet-100
Benchmarks: SVHN CIFAR-10 CIFAR-100 ImageNet-100
Metrics: normalized mutual information accuracy (linear probe) singular value distribution intracluster entropy via GMM
Baselines: Barlow Twins SwAV GEDI GEDI no gen Self-Classifier DINO
Builds on: Barlow Twins SwAV GEDI DINO
Closest papers by meaning
- Bridging the Gap from Asymmetry Tricks to Decorrelation Principles in Non-contrastive Self-supervised Learning (NeurIPS 2022 · similarity 0.90)
- On the duality between contrastive and non-contrastive self-supervised learning (ICLR 2023 · similarity 0.89)
- Understanding Dimensional Collapse in Contrastive Self-supervised Learning (ICLR 2022 · similarity 0.89)
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.88)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.