Research map / Label-efficient and robust learning / Self-supervised and contrastive
LDReg: Local Dimensionality Regularized Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LDReg, a regularizer that increases local intrinsic dimensionality of SSL representations using an asymptotic Fisher-Rao metric, improving SimCLR, SimCLR-Tuned, BYOL, and MAE on ImageNet and transfer tasks.
Problem
Self-supervised representations suffer dimensional collapse; prior work only measured and regularized it globally (via covariance/effective rank), missing local collapse around individual samples.
Approach
Derives an asymptotic Fisher-Rao distance between local nearest-neighbor distance distributions, showing LID comparisons should be done in log scale with the geometric mean, and adds a regularizer that maximizes the mean log-LID of each sample's encoder representations to any SSL loss.
Key results
- On ImageNet with ResNet-50 at 100 epochs, LDReg improves BYOL linear evaluation from 67.6% to 68.5% and SimCLR from 64.3% to 64.8%.
- For MAE with ViT-B, LDReg raises linear evaluation from 57.0% to 57.6% and effective rank from 86.4 to 154.1.
- On COCO with ResNet-50, LDReg raises BYOL AP^bb from 36.30 to 36.82 and AP^mk from 32.17 to 32.47.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LDReg to SimCLR | SimCLR with ResNet-50 · ImageNet linear evaluation, 100 epochs | accuracy (%): 64.3 -> 64.8 | Helped |
| add LDReg to SimCLR-Tuned | SimCLR-Tuned with ResNet-50 · ImageNet linear evaluation, 100 epochs | accuracy (%): 67.2 -> 67.5 | Helped |
| add LDReg to BYOL | BYOL with ResNet-50 · ImageNet linear evaluation, 100 epochs | accuracy (%): 67.6 -> 68.5 | Helped |
| add LDReg to SimCLR | SimCLR with ViT-B · ImageNet linear evaluation, 200 epochs | accuracy (%): 72.9 -> 73.0 | Helped |
| add LDReg to MAE | MAE with ViT-B · ImageNet linear evaluation, 200 epochs | accuracy (%): 57.0 -> 57.6 | Helped |
| add LDReg to SimCLR | SimCLR with ResNet-50 · CIFAR-100 transfer, batch size 2048, 100 epochs | accuracy (%): 71.2 -> 70.6 | Hurt |
| add LDReg to SimCLR | SimCLR with ResNet-50 · DTD transfer, batch size 2048, 100 epochs | accuracy (%): 67.8 -> 67.7 | Hurt |
| add LDReg to SimCLR | SimCLR with ResNet-50 · transfer learning, batch size 2048, 100 epochs | accuracy (%): Food-101: 69.0 -> 69.1; CIFAR-10: 89.1 -> 89.2; Birdsnap: 32.0 -> 33.4; Cars: 36.7 -> 37.3 | Mixed |
| add LDReg to SimCLR | SimCLR with ResNet-50 · transfer learning, batch size 4096, 1000 epochs | accuracy (%): ImageNet: 69.0 -> 69.8; Food-101: 71.1 -> 73.3; CIFAR-10: 90.1 -> 91.8; CIFAR-100: 71.6 -> 75.1; Birdsnap: 37.5 -> 38.7; | Helped |
| use L2 regularization instead of L1 regularization (ablation) | LDReg · SSL experiments | linear evaluation performance | No clear effect |
Surprising findings
- BYOL, which avoids global collapse, has much lower local LID than other methods, indicating local collapse despite high effective rank.
- Effective rank correlates poorly with accuracy across different SSL methods, though it tracks quality within the same method and architecture.
Limitations
- Gains are modest (often below 1 percentage point on ImageNet linear evaluation).
- Transfer-learning baselines were not fully hyperparameter-tuned and reproduced results are slightly below reported values.
- Effective rank is an imperfect cross-method indicator of representation quality.
Experimental setup
Datasets: ImageNet COCO Food-101 CIFAR-10 CIFAR-100 Birdsnap Stanford Cars DTD
Benchmarks: ImageNet linear evaluation Food-101 CIFAR-10 CIFAR-100 Birdsnap Stanford Cars DTD COCO object detection and instance segmentation
Metrics: top-1 linear accuracy effective rank geometric mean of LID AP bb AP mk
Baselines: SimCLR SimCLR-Tuned BYOL MAE
Builds on: SimCLR BYOL MAE LID Representation Theorem method of moments LID estimator
Closest papers by meaning
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.89)
- T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning (NeurIPS 2025 · similarity 0.87)
- Clustering Properties of Self-Supervised Learning (ICML 2025 · similarity 0.85)
- Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.84)
- Exploring Structural Degradation in Dense Representations for Self-supervised Learning (NeurIPS 2025 · similarity 0.84)
- Self-Supervised Representation Learning with Meta Comprehensive Regularization (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.