Research map / Label-efficient and robust learning / Self-supervised and contrastive
A Theoretical Analysis of Self-Supervised Learning for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes GD training of one-layer softmax ViTs under masked reconstruction (MAE) vs contrastive learning on a feature-imbalanced spatial data model, showing MAE learns diverse local attention while CL collapses to global attention.
Problem
Why MAE and contrastive learning produce different attention patterns (diverse local vs. global) in vision transformers, which prior SSL theory did not characterize for transformers.
Approach
Models images as clusters of global and local feature areas with an information gap Delta, and tracks feature-position and position-position attention correlations under gradient descent for one-layer softmax ViTs, proving MAE learns area-wide local attention while CL collapses to global attention even under mild imbalance.
Key results
- MAE-trained one-layer ViT converges to near-optimal reconstruction with area-wide attention for masked patches in both global and local areas, for any information gap.
- CL-trained ViT concentrates attention on the global area for all patches even under mild imbalance (1 - Attn to global area = 1/poly(P)).
- Experiments on ViT-B/16 pretrained on ImageNet-1K show MAE has higher attention diversity than MoCo v3, DINO, and DeiT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use masked reconstruction (MAE) instead of contrastive learning (CL) | One-layer softmax ViT · Theoretical spatial-feature distribution | attention locality and diversity | Helped |
| Train with contrastive learning rather than masked reconstruction | One-layer softmax ViT · Theoretical spatial-feature distribution | attention locality | Hurt |
| Pretrain with MAE rather than MoCo v3, DINO, or DeiT | ViT-B/16 · ImageNet-1K pretraining; attention diversity metric | attention diversity | Helped |
Surprising findings
- Position-position correlations stay near zero for MAE; learning relies on feature-position correlations, contrary to prior positional-attention-only theory.
Limitations
- Analysis restricted to one-layer softmax attention with fixed value matrix and simplified query-key parameterization.
- Assumes information gap magnitude |Delta| = Omega(1) and K = Theta(1) clusters.
- Contrastive analysis requires noisy data variant and temperature/noise specific choices.
Experimental setup
Datasets: ImageNet-1K synthetic clustered spatial data model
Benchmarks: ImageNet-1K attention diversity analysis
Metrics: attention diversity metric reconstruction loss attention concentration
Baselines: MAE MoCo v3 DINO DeiT
Builds on: masked autoencoders MoCo v3 DINO theory of ViTs in supervised learning (Jelassi et al.)
Closest papers by meaning
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.88)
- Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · similarity 0.88)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.88)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.88)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.88)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.