Research map / Label-efficient and robust learning / Self-supervised and contrastive
Exploring Structural Degradation in Dense Representations for Self-supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies Self-supervised Dense Degradation (SDD), where dense prediction performance drops late in SSL training, and proposes the Dense representation Structure Estimator (DSE) for label-free checkpoint selection and as a regularizer.
Problem
Longer self-supervised pretraining can hurt dense prediction performance (e.g., semantic segmentation), and existing unsupervised metrics for transferability do not predict dense performance, so the best dense checkpoint cannot be identified without labels.
Approach
A label-free metric (DSE) combining a class-wise separability measure (inter-class distance minus an intra-class radius estimated from singular values of k-means clusters) with an effective-rank dimensionality measure; it is derived from an error-rate bound for nearest-neighbor classification and used both for checkpoint selection and as a differentiable regularizer.
Key results
- Model selection with DSE improves mIoU by 3.0% on average across sixteen SSL methods and four segmentation benchmarks at roughly 0.01x the GPU cost of supervised selection.
- DSE achieves an average Kendall's tau of 0.57 with downstream dense performance, versus 0.24 for adapted RankMe and 0.33 for adapted Lidar.
- DSE regularization reverses the dense degradation trend for iBOT and I-JEPA, per the paper's figure.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select checkpoints using DSE rather than using the final checkpoint | Sixteen SSL methods · Four dense prediction benchmarks | mIoU: +3.0% | Helped |
| Select checkpoints using training loss instead of DSE | SSL checkpoint selection · Four benchmarks | Delta mIoU: -1.0 | Hurt |
| Select checkpoints using DSE rather than supervised downstream performance | SSL checkpoint selection · Four benchmarks | Delta mIoU: +3.0 vs +3.6 | Mixed |
| Use DSE instead of dense-adapted Lidar to estimate downstream quality | Dense representation transferability estimation · COCO | Kendall's tau: 0.38 -> 0.58 | Helped |
| Use DSE instead of dense-adapted Lidar to estimate downstream quality | Dense representation transferability estimation · Average across benchmarks | Kendall's tau: 0.33 -> 0.57 | Helped |
| Continue DINO training to the final checkpoint rather than selecting an earlier checkpoint | DINO · COCO | mIoU: 4.0% | Hurt |
| Add DSE as a regularizer to the original SSL loss | iBOT and I-JEPA · Dense prediction | downstream performance | Helped |
Surprising findings
- Dense performance degrades during SSL training even when training and evaluation are on the same dataset (COCO), indicating the effect is not overfitting.
- Existing image-level unsupervised transferability estimators (alpha-ReQ, RankMe, Lidar) are negatively or weakly correlated with dense performance.
Limitations
- Theory relies on sub-Gaussian class assumptions and a nearest-neighbor classifier; the bound is for linear-probe-like error rather than full segmentation.
- Pseudo-labels from k-means are used, so the estimate depends on the chosen number of clusters k.
- No error bars reported due to compute cost; significance is tested only via Kendall's tau hypothesis tests.
Experimental setup
Datasets: ImageNet-1k COCO-Stuff PASCAL VOC ADE20k Cityscapes
Benchmarks: COCO-Stuff PASCAL VOC ADE20k Cityscapes
Metrics: mIoU accuracy Kendall's tau GPU hours
Baselines: alpha-ReQ RankMe Lidar training loss selection supervised downstream selection
Builds on: RankMe Lidar alpha-ReQ iBOT I-JEPA
Closest papers by meaning
- Clustering Properties of Self-Supervised Learning (ICML 2025 · similarity 0.86)
- Self-supervised Representation Learning from Random Data Projectors (ICLR 2024 · similarity 0.86)
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.85)
- Self-Supervised Representation Learning with Meta Comprehensive Regularization (AAAI 2024 · similarity 0.85)
- Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning (NeurIPS 2025 · similarity 0.85)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.