Research map / Label-efficient and robust learning / Self-supervised and contrastive
FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FastDINOv2, a two-stage DINOv2 pretraining curriculum that trains first on downsampled low-frequency images and then on full-resolution images with Gaussian noise patching, cutting pretraining cost while keeping ImageNet-C robustness and linear probing accuracy.
Problem
Large-scale self-supervised vision foundation models like DINOv2 are expensive to pretrain, and their robustness to common corruptions typically emerges only at extreme scale; the paper asks whether compute-efficient pretraining can keep both efficiency and robustness.
Approach
Apply a low-frequency data curriculum to DINOv2 (downsampled 112/48 crops for the first 75% of epochs, then full resolution), restart the Adam state at the stage switch, and add Gaussian noise patching in the second stage to offset the high-frequency bias the curriculum induces.
Key results
- On ImageNet-1K with ViT-B/16, FastDINOv2 reduces training time from 16.64 to 10.32 L40S days (about 1.6x) and GFLOPs from 493.76 to 219.92 (2.25x), with ImageNet-1K linear probing of 76.2% vs 77.8% and ImageNet-C error 56.7% vs 56.5% (as reported in the table).
- On ImageNet-100 with ViT-S/16, the 112-224 curriculum reaches 78.44% linear probing in 200 epochs versus 78.6% for the 250-epoch DINOv2 baseline, a 1.44x time speedup.
- Adding Gaussian noise patching raises ImageNet-100-C corruption accuracy from 46.84% to 52.88% while clean accuracy drops from 78.6% to 78.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the 112-224 curriculum with Gaussian noise patching | DINOv2 ViT-B/16 · ImageNet-1K pretraining | training time: 16.64 (NVIDIA L40S) -> 10.32 (NVIDIA L40S) | Helped |
| Apply the 112-224 curriculum with Gaussian noise patching | DINOv2 ViT-B/16 · ImageNet-1K pretraining | GFLOPs: 493.76 -> 219.92 | Helped |
| Apply the 112-224 curriculum with Gaussian noise patching | DINOv2 ViT-B/16 · ImageNet-C | error rate: 56.5% -> 56.7% | No clear effect |
| Apply the 112-224 curriculum with Gaussian noise patching | DINOv2 · ImageNet-100-C corruption accuracy | corruption accuracy: 46.84% -> 52.88% (+6.04%) | Helped |
| Use 112-224 curriculum without Gaussian noise patching | DINOv2 ViT-S/16 · ImageNet-100 | linear probing accuracy: 78.6% -> 78.44% | No clear effect |
| Use 112-224 curriculum with Gaussian noise patching | DINOv2 · ImageNet-1K val | accuracy: 77.8% -> 76.2% | Hurt |
| Use the 75/25 stage split instead of 65/35 or 85/15 (ablation) | FastDINOv2 · ImageNet-100 | accuracy: 77.6% -> 78.1%; 76.8% -> 78.1% | Helped |
| Replace the 112×112 first-stage resolution with 64×64 (ablation) | FastDINOv2 w/o GP · ImageNet-100 | linear probing accuracy: 78.44% -> 70.6% | Hurt |
| Increase stage-one batch size from 40 to 60 and scale the learning rate (ablation) | FastDINOv2 w/o GP · ImageNet-100 | linear probing accuracy: 78.42% -> 78.06% | Hurt |
| Interpolate positional embeddings using 112×112 resolution (ablation) | FastDINOv2 w/o GP · ImageNet-100 | linear probing accuracy: 78.42% -> 78.8% | Helped |
| Interpolate positional embeddings using 224×224 resolution instead of 112×112 (ablation) | FastDINOv2 w/o GP · ImageNet-100 | linear probing accuracy: 78.8% -> 77% | Hurt |
| Train separate positional embeddings instead of reusing them (ablation) | FastDINOv2 w/o GP · ImageNet-100 | linear probing accuracy: 78.8% -> 78.42% | Hurt |
Surprising findings
- The low-frequency curriculum induces a high-frequency bias, improving robustness to low-frequency corruptions (brightness, contrast, fog, frost) while hurting high-frequency noise robustness.
- Gaussian noise patching alone yields a low-frequency bias, improving noise robustness, which is opposite to the curriculum's effect and is used to balance it.
Limitations
- The transition point from low- to full-resolution is a fixed heuristic (75/25) that may not generalize across settings.
- Robustness to medium-frequency corruptions (e.g., blur, pixelate) is mixed, with some drops.
- Main experiments are mostly on ImageNet-100 with ViT-S/16; ImageNet-1K results use a shorter schedule and show a small clean-accuracy drop.
Experimental setup
Datasets: ImageNet-100 ImageNet-1K ImageNet-C ImageNet-100-C ADE20K Oxford Paris
Benchmarks: ImageNet-1K ImageNet-100 ImageNet-C ImageNet-100-C ADE20K Oxford Paris
Metrics: linear probing accuracy corruption accuracy training time GFLOPs mIoU mAcc mAP
Baselines: DINOv2 SimCLR EfficientTrain++
Builds on: DINOv2 EfficientTrain++ RECLIP Gaussian noise augmentation
Closest papers by meaning
- Simplifying DINO via Coding Rate Regularization (ICML 2025 · similarity 0.86)
- Accelerating Augmentation Invariance Pretraining (NeurIPS 2024 · similarity 0.85)
- Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness (ICLR 2025 · similarity 0.85)
- Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks (NeurIPS 2025 · similarity 0.84)
- Tuning Frequency Bias of State Space Models (ICLR 2025 · similarity 0.84)
- Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.