Research map / Label-efficient and robust learning / Self-supervised and contrastive
Modulate Your Spectrum in Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spectral Transformation (ST), a framework generalizing whitening loss, and IterNorm with trace loss (INTL) for self-supervised learning that avoids dimensional collapse.
Problem
Dimensional collapse in joint-embedding self-supervised learning, where embedding covariance develops zero eigenvalues; whitening-based losses prevent it but hard whitening is numerically unstable and whether other functions can avoid collapse is unclear.
Approach
Generalizes whitening into spectral transformation g(Σ) that maps the embedding spectrum; shows IterNorm (Newton iteration) is an instance of ST, and adds a trace loss on the transformed output so the spectrum is driven toward equal eigenvalues, with a theoretical convexity argument.
Key results
- INTL reaches 92.60% top-1 on CIFAR-10, 70.88% on CIFAR-100 and 81.68% on ImageNet-100 with ResNet-18 linear evaluation, on par with or above reproduced baselines.
- On ImageNet with ResNet-50 and batch size 512, INTL gets 69.5% top-1 at 100 epochs and 73.1% at 800 epochs; with EMA it reaches 74.3% at 800 epochs.
- INTL transfers to COCO detection with 40.7 AP and instance segmentation with 35.4 AP, outperforming the listed baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add trace loss to IterNorm (ablation) | IterNorm on CIFAR-10 · CIFAR-10 | top-1 accuracy | Helped |
| remove IterNorm while retaining trace loss (ablation) | INTL on CIFAR-10 · CIFAR-10 | top-1 accuracy: 16.15% | Hurt |
| use EMA with INTL (ablation) | INTL on ImageNet · ImageNet, 800 epochs | top-1 accuracy: 73.1 -> 74.3 | No clear effect |
| increase projection dimension from 2048 to 8192 (ablation) | INTL on ImageNet · ImageNet linear evaluation | top-1 accuracy: +0.14% | Helped |
| add multi-crop augmentation (ablation) | INTL on ImageNet · ImageNet linear evaluation | top-1 accuracy: 76.5% -> 76.6% | Helped |
| increase IterNorm iteration count to T=9 (ablation) | IterNorm without trace loss · CIFAR-10 | maximum eigenvalue of whitened output: around 10^7 | Hurt |
| increase IterNorm iteration count to T≥11 (ablation) | IterNorm without trace loss · CIFAR-10 | trainability: T\geq11; NAN | Hurt |
| set power-function exponent p near 0.5 (0.45 or 0.55) instead of deviating from 0.5 (ablation) | Spectral Transformation · real-world datasets | evaluation performance | Helped |
| increase embedding dimension beyond batch size when using hard whitening (ablation) | Hard whitening · ImageNet and CIFAR-10 | training stability: d= 1024 or 2048; 10^{-12} \sim 10^{-10} | Hurt |
Surprising findings
- IterNorm alone fails in SSL: numerical divergence occurs with large iteration counts (NaN loss for T>=11), and trace loss alone achieves only 16.15% top-1 on CIFAR-10.
- Power-function spectral transformations near p=0.5 (not only exact whitening) also avoid collapse, but training crashes when embedding dimension is large relative to batch size.
Limitations
- IterNorm without trace loss suffers numerical divergence and collapse for large iteration counts T
- Hard whitening via eigen-decomposition is numerically unstable, and the trace loss weight beta relies on a batch-size-dependent heuristic
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet COCO2017
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet COCO detection COCO instance segmentation
Metrics: top-1 linear accuracy 5-nearest neighbors accuracy AP AP50 AP75
Baselines: SimCLR MoCo v2 BYOL SwAV SimSiam W-MSE Shuffled-DBN Barlow Twins VICReg Zero-CL CW-RGP DINO
Builds on: W-MSE IterNorm Barlow Twins VICReg BYOL
Closest papers by meaning
- Self-supervised Transformation Learning for Equivariant Representations (NeurIPS 2024 · similarity 0.86)
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.85)
- Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.85)
- Spectral Reversal: Counteracting Singular Value Bias for Graph Prompting (NeurIPS 2026 · similarity 0.84)
- STNet: Spectral Transformation Network for Solving Operator Eigenvalue Problem (NeurIPS 2025 · similarity 0.84)
- I-Con: A Unifying Framework for Representation Learning (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.