Research map / Label-efficient and robust learning / Self-supervised and contrastive

Simplifying DINO via Coding Rate Regularization

ICML 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Invariance-based self-supervised learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SimDINO and SimDINOv2, simplified versions of DINO and DINOv2 that replace the heads, centering, softmax and cross-entropy machinery with a squared Euclidean alignment loss plus a coding rate regularizer, yielding more stable training and better ImageNet, segmentation and video results.

Problem

DINO and DINOv2 self-supervised pipelines are complex, hyperparameter-sensitive and prone to representation collapse, making them hard to improve or adapt to new domains.

Approach

Replace DINO's empirically designed heads, centering, temperature softmax and cross-entropy with a direct squared Euclidean distance between normalized student and teacher CLS (and patch) features, and prevent collapse with an explicit total coding rate regularizer on the student feature covariance.

Key results

• SimDINO ViT-B/16 reaches 74.9 k-NN and 77.3 linear on ImageNet-1K at 100 epochs, versus 72.9 and 76.3 for DINO.
• SimDINOv2 ViT-B/16 improves DINOv2 on ImageNet-1K k-NN from 76.0 to 78.1 and on ADE20K linear mIoU from 32.5 to 36.9.
• DINO ViT-L/16 training diverged under the ViT-B hyperparameters, while SimDINO ViT-L/16 reached 75.6 k-NN and 77.4 linear.

Measured effects reported in the paper

Surprising findings

• DINO diverges (NaN) under small changes to momentum, last-layer normalization, or teacher temperature, showing high sensitivity.
• SimDINO trained without self-distillation (teacher equal to student) reaches 58.6 k-NN and 68.0 linear, whereas DINO collapses in that setting.

Limitations

• The coding rate weight gamma is a heuristic choice that requires constant tuning, per the authors.
• DAVIS video segmentation results for DINOv2 variants underperform DINO, attributed to evaluation sensitivity and not resolved.
• Object detection/segmentation results via MaskCut are low in absolute terms (AP around 2).

Experimental setup

Datasets: ImageNet-1K COCO train2017

Benchmarks: ImageNet-1K COCO val2017 ADE20K DAVIS-2017

Metrics: k-NN accuracy linear accuracy AP50 AP75 AP mIoU mAcc J&F mean J mean F mean

Baselines: DINO DINOv2 SwAV MoCov3 MaskCut with DINO ViT-B/8

Builds on: DINO DINOv2 iBOT MMCR coding rate KoLeo estimator

Closest papers by meaning

• FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed (NeurIPS 2025 · similarity 0.86)
• Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency (CVPR 2025 · similarity 0.84)
• Diversify, Contextualize, and Adapt: Efficient Entropy Modeling for Neural Image Codec (NeurIPS 2024 · similarity 0.84)
• Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation (ICML 2025 · similarity 0.83)
• Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.83)
• RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations (ICLR 2024 · similarity 0.83)