Research map / Label-efficient and robust learning / Self-supervised and contrastive
What shapes the loss landscape of self supervised learning?
Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, Hidenori Tanaka
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add normalization (VICReg-style variance regularization, infinite kappa) to linear SSL landscape (ablation) | linear SSL model with InfoNCE/Barlow Twins-type losses · theory | collapse occurrence | Mixed |
| Add a bias term to a normalized linear model (ablation) | normalized linear SSL model · theory | collapse occurrence | Hurt |
| Add weight decay to the linear SSL model (ablation) | linear SSL model · theory | collapse occurrence | Hurt |
| Use beta-InfoNCE with beta<1 (weight entropy term) to control collapse (ablation) | InfoNCE · theory and numerical simulation | degree of dimensional collapse | Improved |
| Use task-aligned (style-targeting) augmentation instead of isotropic augmentation with beta-InfoNCE (ablation) | beta-InfoNCE (beta=1/2) on synthetic regression · synthetic structured data | downstream generalization error | Improved |
| Use spectral contrastive loss instead of SimCLR/InfoNCE | ResNet-12 on imbalanced CIFAR-10 · imbalanced CIFAR-10 | robustness to data imbalance | Improved |
| Use a large sample size N in NT-xent / weighted InfoNCE (collapse depends on N) (ablation) | weighted InfoNCE (alpha=0.1) · linear model, Gaussian data | smallest eigenvalues of W^TW | Improved |
| Apply standard feature normalization f(x)/||f(x)|| in the linear model (ablation) | linear SSL model · linear model numerical experiment | smallest eigenvalues of W^TW | Hurt |
Closest papers by meaning
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.89)
- Exploring Structural Degradation in Dense Representations for Self-supervised Learning (NeurIPS 2025 · similarity 0.88)
- Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.88)
- Clustering Properties of Self-Supervised Learning (ICML 2025 · similarity 0.87)
- LDReg: Local Dimensionality Regularized Self-Supervised Learning (ICLR 2024 · similarity 0.86)
- Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss (Student Abstract) (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
