Research map / Label-efficient and robust learning / Self-supervised and contrastive

Can Generative Models Improve Self-Supervised Representation Learning?

AAAI 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Invariance-based self-supervised learning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adds instance-conditioned generative augmentations (Stable Diffusion via unCLIP-style CLIP conditioning, or ICGAN) to joint-embedding SSL pipelines, pre-generating 10 samples per ImageNet image and gaining up to ~10% linear-probe Top-1.

Problem

Standard SSL augmentations (crop, color jitter, blur, flip) produce limited diversity and cannot capture real-world variation, which constrains representation quality.

Approach

Use pretrained instance-conditioned generative models (Stable Diffusion conditioned on a CLIP image embedding, or ICGAN) to produce semantically consistent synthetic views of a source image, prepended to standard augmentations in joint-embedding SSL, with the generator kept frozen.

Key results

• Stable Diffusion generative augmentation improves SimCLR/MoCo/BYOL/SimSiam/Barlow Twins linear probe Top-1 on ImageNet by roughly 3% to 10% over baseline (e.g., MoCo 60.23 to 70.17 Top-5-matched table; SimCLR 61.84 to 68.12).
• ICGAN gives smaller gains and slightly hurts Barlow Twins on ImageNet (-1.2 Top-1 on Food101 and -2.15 on CIFAR10 for Barlow Twins).
• Generative augmentation alone (no standard augmentations) gives 63.80 vs 61.84 baseline on ImageNet, but adding standard augmentations on top gives the best result (68.11).

Measured effects reported in the paper

Surprising findings

• Replacing standard augmentations with generative ones alone still gives a gain, but combining both is best; the authors found standard augmentations cannot be dropped.
• ICGAN-based augmentations degrade Barlow Twins on several datasets despite improving the other SSL methods.

Limitations

• Relies on pretrained generative models that are not co-trained with the SSL encoder.
• Offline generation of augmentations is computationally expensive (about 178 GPU-hours on A100 for Stable Diffusion over ImageNet).
• Gains from ICGAN are inconsistent, and the Stable Diffusion conditioning encoder (CLIP, trained on 400M pairs) provides outside data advantage.

Experimental setup

Datasets: ImageNet Food101 CIFAR10 CIFAR100 Places365 iNaturalist 2018

Benchmarks: ImageNet Food101 CIFAR10 CIFAR100 Places365 iNaturalist 2018

Metrics: Top-1 accuracy Top-5 accuracy CKA dissimilarity Orthogonal Procrustes distance

Baselines: Baseline standard augmentations ICGAN augmentation StableRep

Builds on: SimCLR MoCo BYOL SimSiam Barlow Twins ICGAN Stable Diffusion solo-learn

Closest papers by meaning

• Generate in Reconstruction Space, Match in Semantic Space: Transport Geometry for One-Step Generation (NeurIPS 2026 · similarity 0.89)
• You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning (NeurIPS 2024 · similarity 0.88)
• Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.88)
• Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation (ICLR 2024 · similarity 0.88)
• Return of Unconditional Generation: A Self-supervised Representation Generation Method (NeurIPS 2024 · similarity 0.88)
• Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.88)