Research map / Label-efficient and robust learning / Self-supervised and contrastive

Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization

NeurIPS 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Data-efficient and robust self-supervision · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes orthogonality regularization (SO and SRIP) on encoder conv and linear weights during SSL pretraining to prevent dimensional collapse of weights, hidden features, and representations, showing gains across 13 SSL methods and CNN/ViT backbones.

Problem

Dimensional collapse in self-supervised learning, where a few large eigenvalues dominate, affects not only representations but also weight matrices and hidden features of the encoder, which existing representation-level fixes do not address.

Approach

Add an orthogonality regularizer (Soft Orthogonality Frobenius-norm or Spectral Restricted Isometry Property spectral-norm penalty on Gram matrix minus identity) to all conv and linear weights of the encoder during SSL pretraining, which keeps weight filters decorrelated and, by a proposition on orthogonal maps, preserves feature normalization and decorrelation across layers.

Key results

• On CIFAR-100 with ResNet18, SRIP raises DINO top-1 from 59.13 to 62.75 and SO/SRIP raise MOCOv3 from 67.13 to 69.37 (SRIP).
• BYOL pretrained on ImageNet-1k (ResNet50, 100 epochs) improves top-1 from 65.81 to 67.91 with SRIP and 67.84 with SO.
• On VOC object detection, BYOL with SO improves AP from 44.74 to 53.81 over BYOL without OR.

Measured effects reported in the paper

Surprising findings

• Removing the VICReg projector (whitening representations directly) accelerated eigenvalue decay of weight matrices and lowered downstream accuracy, so representation-level whitening does not fix weight-level collapse.
• Gains are larger for ViT backbones (up to 12% top-1) than for CNNs, which the authors attribute to linear layers having more redundant filters.

Limitations

• Experiments use solo-learn and LightlySSL setups rather than official SSL implementations (e.g., no multi-crop in DINO), and the authors leave migration to official code to future work.
• Generative SSL methods (e.g., MAE) were not studied.
• The regularization weight gamma must be tuned per backbone and regularizer on a validation set.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet-1k Food-101 Flowers-102 DTD GTSRB PASCAL VOC 2007 PASCAL VOC 2012

Benchmarks: CIFAR-100 CIFAR-10 ImageNet-1k ImageNet-100 Food-101 Flowers-102 DTD GTSRB PASCAL VOC 2007+2012 object detection

Metrics: Top-1 accuracy Top-5 accuracy AP AP50 AP75

Baselines: VICReg feature whitening VICReg without projector BYOL without OR MOCOv2plus MOCOv3 DINO NNBYOL BYOL Barlow Twins SimCLR SwaV Scratch

Builds on: BYOL VICReg Solo-learn Spectral Restricted Isometry Property regularization DINO

Closest papers by meaning

• LDReg: Local Dimensionality Regularized Self-Supervised Learning (ICLR 2024 · similarity 0.89)
• Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.88)
• Exploring Structural Degradation in Dense Representations for Self-supervised Learning (NeurIPS 2025 · similarity 0.85)
• T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning (NeurIPS 2025 · similarity 0.85)
• Modulate Your Spectrum in Self-Supervised Learning (ICLR 2024 · similarity 0.85)
• Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.85)