Research map / Label-efficient and robust learning / Self-supervised and contrastive

You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning

NeurIPS 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Data-efficient and robust self-supervision

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that DINOv2 joint-embedding SSL can reach state-of-the-art results with only random cropping (no resizing, no photometric augmentations) when pretrained on sufficiently large data, and that augmentation mainly acts by increasing effective dataset size.

Problem

Joint-embedding architecture SSL methods are believed to require hand-crafted data augmentations for invariance learning and strong performance; the paper asks whether these augmentations are necessary at scale.

Approach

Case study on DINOv2 varying augmentation configurations (Original, Shared, Crop+Resize, Crop) across ImageNet-1k, ImageNet-22k and LVD-142M and model sizes; finds that with enough data, cropping without resizing plus masking matches augmented training, and that augmentation's main role is artificially increasing dataset size rather than enforcing invariance.

Key results

• Crop-only DINOv2 ViT-L pretrained on ImageNet-22k reaches 84.0% ImageNet-1k linear val accuracy and 43.5 ADE20k mIoU, outperforming MAE, BEiT, I-JEPA and AIM in the compared table.
• Gap between Shared and Original augmentation setups on ImageNet-1k linear probe shrinks from -1.3% (ImageNet-1k) to -0.4% (LVD-142M) as data grows.
• With High Compute hyperparameters on ImageNet-1k, Crop+Resize reaches only 46% linear accuracy, whereas tuned low-compute hyperparameters reach 73.5%.

Measured effects reported in the paper

Surprising findings

• Hyperparameters tuned for high compute on ImageNet-1k make DINOv2 without augmentations fail (46% linear accuracy), so conclusions depend on the chosen compute regime.
• Enforcing invariance via augmentation becomes less useful at scale: the Shared setup, which does not enforce invariance, nearly matches Original.

Limitations

• Only the DINOv2 algorithm was studied; results may differ for other SSL methods.
• A small performance gap remains between augmented and non-augmented setups for large architectures with long training.
• The mechanism by which augmentations ease optimization is not understood.

Experimental setup

Datasets: ImageNet-1k ImageNet-22k LVD-142M Places205 iNaturalist18 ImageNet-v2 ImageNet-Real ADE20k NYU-Depth v2 FlickrLogos32 GTSRB RP2K

Benchmarks: ImageNet-1k ImageNet-v2 ImageNet-Real Places205 iNaturalist18 ADE20k NYU-Depth v2 FlickrLogos32 GTSRB RP2K Products-10k FireRisk

Metrics: linear probe accuracy mIoU RMSE AUROC cosine similarity

Baselines: AIM BEiT MAE I-JEPA DINOv2 Original augmentation setup

Builds on: DINOv2 DINO iBOT I-JEPA SimCLR

Closest papers by meaning

• Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.88)
• How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks (NeurIPS 2024 · similarity 0.88)
• Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning (NeurIPS 2025 · similarity 0.87)
• Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning (NeurIPS 2024 · similarity 0.87)
• T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data (ICLR 2025 · similarity 0.86)
• Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression (ICLR 2024 · similarity 0.86)