Research map / Label-efficient and robust learning / Self-supervised and contrastive
On the Importance of Embedding Norms in Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that embedding norms slow SSL convergence quadratically, encode model confidence, and can be controlled via weight decay, cut-initialization, and a GradScale gradient-rescaling layer.
Problem
Cosine-similarity-based SSL methods seemingly ignore embedding norms, yet norms affect gradient magnitudes, convergence speed, and carry confidence information; the interaction was not systematically understood.
Approach
Proves that SSL gradients scale inversely with embedding norm and that gradient steps grow norms, causing a quadratic slowdown ("embedding-norm effect"); proposes interventions (weight decay, cut-initialization dividing weights by c, GradScale layer with power p) to mitigate it, and shows norms encode novelty, accuracy, and human confidence.
Key results
- Cut-initialization with tuned weight decay improves SimSiam kNN accuracy on ImageNet-100 from 62.0 to 67.2 at 500 epochs and on class-imbalanced CIFAR-10 from 47.0 to 61.7.
- GradScale improves SimCLR kNN accuracy on CIFAR-100 from 56.6 to 58.2 at 1000 epochs but fails to converge on ImageNet-100 (1.0%).
- Simulations show convergence slowdown scales quadratically with embedding norm, and embedding norms correlate monotonically with kNN accuracy and human label agreement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cut-initialization (c=3) (ablation) | SimCLR · Cifar-100 | kNN accuracy: 56.6 -> 60.1 | Helped |
| add cut-initialization (c=3) (ablation) | SimCLR · ImageNet-100 | kNN accuracy: 59.4 -> 60.9 | Helped |
| add GradScale (power p=1) (ablation) | SimCLR · Cifar-100 | kNN accuracy: 56.6 -> 58.2 | Helped |
| add GradScale (power p=1) (ablation) | SimCLR · ImageNet-100 | kNN accuracy: 59.4 -> 01.0 | Hurt |
| add cut-initialization (c=9) (ablation) | SimSiam · ImageNet-100 | kNN accuracy: 62.0 -> 67.2 | Helped |
| add cut-initialization (c=3) (ablation) | SimCLR · Cifar-10 imbalanced | kNN accuracy: 56.5 -> 61.1 | Helped |
| add GradScale (power p=1) (ablation) | SimCLR · Cifar-100 imbalanced | kNN accuracy: 24.3 -> 29.1 | Helped |
| add cut-initialization (c=3) (ablation) | MoCo V2 · ImageNet-100 | Top-1 accuracy: 38.7 -> 43.8 | Helped |
| add cut-initialization (c=3) (ablation) | Dino · ImageNet-100 | Top-1 accuracy: 43.7 -> 29.0 | Hurt |
| use weight decay γ=10.0 (ablation) | simulated cosine-similarity training · embedding-pair convergence simulation | steps to convergence: divergence | Hurt |
| increase cut constant from c=1 to c=8 at default weight decay (ablation) | SimSiam · ImageNet-100 | kNN accuracy: 40.5 -> 50.2 | Helped |
| increase cut constant from c=1 to c=8 at default weight decay (ablation) | SimCLR · Cifar-100 | kNN accuracy: 40.5 -> 36.8 | Hurt |
Surprising findings
- GradScale with p=1 failed to converge on ImageNet-100 and for non-contrastive models.
- Embeddings can shrink without weight decay in SimSiam, attributed to shared weights.
Limitations
- GradScale with p=1 is unstable and fails on ImageNet-100 and non-contrastive models.
- Theory and simulations assume plain gradient descent; adaptive optimizer results are empirical only.
- Dino does not benefit from cut-initialization since its objective does not use cosine similarity.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100 Tiny-ImageNet Flowers102 CIFAR-10-N CIFAR-100-N CIFAR-10-H
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100 Tiny-ImageNet Flowers102
Metrics: kNN accuracy linear probe accuracy steps to convergence embedding norm Top-1 accuracy Top-5 accuracy
Baselines: SimCLR SimSiam BYOL MoCo v2 MoCo v3 DINO
Builds on: SimCLR SimSiam BYOL MoCo
Closest papers by meaning
- Clustering Properties of Self-Supervised Learning (ICML 2025 · similarity 0.86)
- Improving Self-Supervised Learning by Characterizing Idealized Representations (NeurIPS 2022 · similarity 0.86)
- Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning (NeurIPS 2025 · similarity 0.86)
- Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data (ICLR 2021 · similarity 0.86)
- How Low Can You Go? Searching for the Intrinsic Dimensionality of Complex Networks using Metric Node Embeddings (ICLR 2025 · similarity 0.85)
- Information Loss and Disparate Effects in Network Embeddings (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.