Research map / Label-efficient and robust learning / Self-supervised and contrastive

Rethinking the Uniformity Metric in Self-Supervised Learning

ICLR 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Invariance-based self-supervised learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies four principled properties for uniformity metrics, shows the Gaussian-potential uniformity loss fails most of them, and proposes a quadratic-Wasserstein-based uniformity loss that satisfies all four and improves SSL methods on CIFAR-10/100.

Problem

Existing uniformity metric for self-supervised representations (log average Gaussian potential) is sensitive to sample replications and fails to capture feature redundancy and dimensional collapse.

Approach

Define four properties (permutation invariance, instance cloning invariance, feature cloning and feature baby strict decrease), and propose a uniformity loss equal to the quadratic Wasserstein distance between a Gaussian fit to the normalized representations and N(0, I/m), which has a closed form via mean and covariance and satisfies all four properties.

Key results

• Adding the Wasserstein uniformity loss to MoCo v2 raises CIFAR-100 Acc@1 from 60.27 to 63.68 (+3.41).
• BYOL + W2 improves CIFAR-10 Acc@1 from 89.53 to 90.31 and CIFAR-100 from 63.66 to 65.16.
• The existing metric -L_U is shown to violate the instance cloning, feature cloning, and feature baby properties, and remains almost unchanged even at 80% zeroed coordinates.

Measured effects reported in the paper

Surprising findings

• The widely used uniformity metric -L_U increases when features are cloned, i.e., it rewards redundancy rather than penalizing it.
• Adding the Wasserstein uniformity loss can slightly compromise alignment while improving uniformity and accuracy.

Limitations

• The four properties may not comprehensively characterize an ideal uniformity metric.
• The metric relies on a Gaussian hypothesis and an asymptotic approximation of the uniform spherical distribution that is only justified for moderately large dimensions (m >= 16).

Experimental setup

Datasets: CIFAR-10 CIFAR-100

Benchmarks: CIFAR-10 CIFAR-100

Metrics: Top-1 accuracy Top-5 accuracy uniformity loss W2 uniformity loss L_U alignment loss

Baselines: SimCLR MoCo v2 NNCLR SimSiam AlignUniform BYOL BarlowTwins Zero-CL

Builds on: alignment and uniformity (Wang & Isola) Barlow Twins VICReg BYOL SimSiam

Closest papers by meaning

• Stabilizing Sample Similarity in Representation via Mitigating Random Consistency (ICML 2025 · similarity 0.84)
• Exploring Structural Degradation in Dense Representations for Self-supervised Learning (NeurIPS 2025 · similarity 0.84)
• Improve Representation for Imbalanced Regression through Geometric Constraints (CVPR 2025 · similarity 0.84)
• Hierarchical Integral Probability Metrics: A distance on random probability measures with low sample complexity (ICML 2024 · similarity 0.83)
• Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.83)
• LDReg: Local Dimensionality Regularized Self-Supervised Learning (ICLR 2024 · similarity 0.83)