Research map / Label-efficient and robust learning / Self-supervised and contrastive
Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives closed-form solutions for linear reconstruction-based and joint-embedding SSL under augmentation-noise alignment, and shows joint-embedding is preferable when irrelevant noise has high magnitude.
Problem
When should practitioners prefer reconstruction-based versus joint-embedding self-supervised learning, and how do data augmentations affect the learned representations in each?
Approach
Closed-form solutions for linear reconstruction and joint-embedding SSL parameterized by the augmentation covariance; shows both need augmentation-noise alignment even with infinite samples, and that joint-embedding imposes a strictly weaker alignment condition when irrelevant noise magnitude is large.
Key results
- Joint-embedding (DINO) drops 10.5% and BYOL 12.4% in average accuracy from corruption severity 1 to 5, versus 25.1% for MAE.
- Theory: for high-magnitude irrelevant noise, the joint-embedding alignment threshold alpha_JE is strictly lower than the reconstruction threshold alpha_RC; for low-magnitude noise the ordering reverses.
- Supervised learning reaches optimal performance for any alignment as n grows, whereas SSL requires alignment even with infinite samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase corruption severity from 1 to 5 | MAE (ViT) · ImageNet-C | top-1 linear probing accuracy drop: 25.1% average drop | Hurt |
| Increase corruption severity from 1 to 5 | DINO (ViT) · ImageNet-C | top-1 linear probing accuracy drop: 10.5% drop | Hurt |
| Increase corruption severity from 1 to 5 | BYOL (ResNet) · ImageNet-C | top-1 linear probing accuracy drop: 12.4% drop | Hurt |
| Use joint-embedding rather than reconstruction | DINO versus MAE · ImageNet-C Pixelate | accuracy drop, Sev. 1 to 5: 27.9 -> 12.4 | Helped |
| Use joint-embedding rather than reconstruction | DINO versus MAE · ImageNet-C Gaussian Noise | accuracy drop, Sev. 1 to 5: 27.3 -> 12.7 | Helped |
| Use joint-embedding rather than reconstruction | DINO versus MAE · ImageNet-C Zoomblur | accuracy drop, Sev. 1 to 5: 20.0 -> 6.5 | Helped |
| Align augmentations with the irrelevant noise | Joint-embedding SSL · Noisy data settings | performance | Helped |
| Increase sample size without improving augmentation-noise alignment | Self-supervised learning · Noisy data, infinite-sample limit | recovery of optimal performance | No clear effect |
| Increase sample size | Supervised learning · Noisy data, infinite-sample limit | recovery of optimal performance | Helped |
| Use reconstruction rather than joint-embedding | SSL methods · Low-magnitude irrelevant noise | augmentation alignment requirement | Helped |
| Use joint-embedding rather than reconstruction | SSL methods · High-magnitude irrelevant noise | augmentation alignment requirement | Helped |
Surprising findings
- Unlike supervised learning, increasing sample size cannot overcome augmentation-noise misalignment in SSL.
- Reconstruction is preferable when irrelevant noise is low-magnitude, contrary to the general preference for joint-embedding.
Limitations
- Analysis restricted to linear models and Gaussian data/augmentations.
- Asymptotic results only; finite sample size characterization left to future work.
- Assumes important and noise components are aligned with a shared eigenbasis.
Experimental setup
Datasets: ImageNet-100 ImageNet-C Fashion-MNIST Kuzushiji-MNIST scRNA-seq synthetic Gaussian data
Benchmarks: ImageNet-100 linear probing ImageNet-C corruptions
Metrics: top-1 linear probing accuracy accuracy drop from severity 1 to 5
Baselines: MAE DINO BYOL SimCLR VICReg
Builds on: VICReg Masked Autoencoder Denoising Autoencoder Barlow Twins
Closest papers by meaning
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.88)
- You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning (NeurIPS 2024 · similarity 0.87)
- Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projections (NeurIPS 2025 · similarity 0.87)
- Latent Functional Maps: a spectral framework for representation alignment (NeurIPS 2024 · similarity 0.86)
- PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment (NeurIPS 2026 · similarity 0.86)
- Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.