Research map / Label-efficient and robust learning / Domain adaptation and generalization
Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Randomized-MLP (RMLP) regularization, a non-trainable random residual operator replacing DINOv2 projection heads, to fine-tune ViTs for better domain adaptation and attention interpretability on medical and natural images.
Problem
ViTs such as DINOv2 repurpose low-information patch tokens as global-context storage, degrading attention interpretability and dense-task performance, especially under medical domain shift.
Approach
Replace the learnable DINO/iBOT MLP heads with randomized, non-trainable Gaussian residual maps (R_lambda-MLP) during contrastive fine-tuning, motivated by a topological/random-projection analysis that the heads should preserve embedding geometry.
Key results
- DINOv2-S + R_5-MLP reaches averaged DICE 0.97±0.03 on OCT retinal layer segmentation vs 0.86±0.20 for DINOv2-S + MLP and 0.79±0.20 for DINOv2-S.
- On ImageNet-1k 1-NN classification, DINOv2-S + R_5-MLP gets 0.65 vs 0.64 for DINOv2-S, and linear head 0.76 vs 0.73 for MLP fine-tuning.
- Patch-token norm correlation with retinal-layer presence rises to 0.21±0.03 with R_5-MLP vs 0.19±0.03 with MLP and -0.13±0.14 for DINOv2-S.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the MLP head with an RMLP at amplitude 5 | DINOv2-S fine-tuning · OCT retinal layer segmentation | DICE: 0.86±0.20 -> 0.97±0.03 | Helped |
| Replace the MLP head with an RMLP at amplitude 5 | DINOv2-S fine-tuning · ImageNet-1k, linear-head semantic segmentation | score: 0.75±0.1 -> 0.77±0.1 | Helped |
| Replace the MLP head with an RMLP at amplitude 5 | DINOv2-S fine-tuning · ImageNet-1k, ViT-UNet depth estimation | error: 8±3 -> 7±3 | Helped |
| Replace the MLP head with an RMLP at amplitude 5 | DINOv2-S fine-tuning · VOC07 object localization | CorLoc: 31.18±0.03 -> 31.17±0.05 | No clear effect |
| Use RMLP amplitude 0.1 instead of amplitude 5 (ablation) | DINOv2 fine-tuning · OCT retinal layer segmentation, averaged DICE | DICE: 0.97±0.03 -> 0.94±0.12 | Hurt |
| Use RMLP amplitude 10 instead of amplitude 5 (ablation) | DINOv2 fine-tuning · OCT retinal layer segmentation, averaged DICE | DICE: 0.97±0.03 -> 0.87±0.20 | Hurt |
| Use RMLP amplitude 20 instead of amplitude 5 (ablation) | DINOv2 fine-tuning · OCT retinal layer segmentation, averaged DICE | DICE: 0.97±0.03 -> 0.70±0.30 | Hurt |
| Use RMLP amplitude 0.1 instead of amplitude 5 (ablation) | SwAV fine-tuning · OCT retinal layer segmentation, averaged DICE | DICE: 0.83±0.27 -> 0.92±0.12 | Helped |
| Use RMLP amplitude 10 instead of amplitude 0.1 (ablation) | SwAV fine-tuning · OCT retinal layer segmentation, averaged DICE | DICE: 0.92±0.12 -> 0.71±0.33 | Hurt |
| Use RMLP amplitude 20 instead of amplitude 0.1 (ablation) | SwAV fine-tuning · OCT retinal layer segmentation, averaged DICE | DICE: 0.92±0.12 -> 0.84±0.26 | Hurt |
| Replace the MLP head with an RMLP at amplitude 0.1 | SwAV fine-tuning · OCT retinal layer segmentation, ONL DICE | DICE: 0.6±0.18 -> 0.64±0.1 | Helped |
Surprising findings
- Linear-head classification with RMLP fine-tuning slightly trails DINOv2-S, which the authors attribute to neighborhood-based versus linearly separable latent structure.
- Larger amplitudes (e.g., R_20) degrade performance, and smaller amplitude (R_0.1) does not consistently improve results, showing a tradeoff in amplitude.
Limitations
- Optimal regularization amplitude depends on modality or latent dimensionality and is chosen heuristically.
- Experiments limited to small- and mid-scale datasets.
- Linear-probe classification with RMLP slightly trails baselines.
Experimental setup
Datasets: ImageNet-1k ADE20k NYU-Depth V2 OCTID Glaucoma Fundus IDRID JSIEC MESSIDOR-2 PAPILA Retina Aptos BSDS300
Benchmarks: ADE20k NYU-Depth V2 ImageNet-1k PASCAL VOC 2007 OCTID Glaucoma Fundus IDRID JSIEC MESSIDOR-2 PAPILA Retina Aptos
Metrics: DICE accuracy CorLoc RMSE-style depth error Pearson correlation
Baselines: DINOv2-S + MLP Registers DVT Sinder RetFound SwAV + MLP
Builds on: DINOv2 iBOT KoLeo regularizer SwAV
Closest papers by meaning
- LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.85)
- MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning (CVPR 2024 · similarity 0.85)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.85)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.84)
- Perturbating, Tuning, and Collaborating: Harnessing Vision Foundation Models for Single Domain Generalization on Medical Imaging (AAAI 2025 · similarity 0.84)
- Learning General-purpose Biomedical Volume Representations using Randomized Synthesis (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.