AI Paper Atlas

Research map / Label-efficient and robust learning / Self-supervised and contrastive

What shapes the loss landscape of self supervised learning?

Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, Hidenori Tanaka

ICLR 2023 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Data-efficient and robust self-supervision · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add normalization (VICReg-style variance regularization, infinite kappa) to linear SSL landscape (ablation)linear SSL model with InfoNCE/Barlow Twins-type losses · theorycollapse occurrenceMixed
Add a bias term to a normalized linear model (ablation)normalized linear SSL model · theorycollapse occurrenceHurt
Add weight decay to the linear SSL model (ablation)linear SSL model · theorycollapse occurrenceHurt
Use beta-InfoNCE with beta<1 (weight entropy term) to control collapse (ablation)InfoNCE · theory and numerical simulationdegree of dimensional collapseImproved
Use task-aligned (style-targeting) augmentation instead of isotropic augmentation with beta-InfoNCE (ablation)beta-InfoNCE (beta=1/2) on synthetic regression · synthetic structured datadownstream generalization errorImproved
Use spectral contrastive loss instead of SimCLR/InfoNCEResNet-12 on imbalanced CIFAR-10 · imbalanced CIFAR-10robustness to data imbalanceImproved
Use a large sample size N in NT-xent / weighted InfoNCE (collapse depends on N) (ablation)weighted InfoNCE (alpha=0.1) · linear model, Gaussian datasmallest eigenvalues of W^TWImproved
Apply standard feature normalization f(x)/||f(x)|| in the linear model (ablation)linear SSL model · linear model numerical experimentsmallest eigenvalues of W^TWHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy