Research map / Label-efficient and robust learning / Self-supervised and contrastive
Information Flow in Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes Barlow Twins, spectral contrastive learning, and MAE/U-MAE through matrix mutual information and matrix joint entropy, then proposes M-MAE, which adds a matrix-based entropy (TCR) regularizer to the MAE loss and reports gains on ImageNet-1K ViT pretraining.
Problem
Understanding how Siamese self-supervised methods (Barlow Twins, spectral contrastive learning) and single-branch masked autoencoders relate under a unified information-theoretic view, and how to improve MAE with that view.
Approach
Uses matrix-based Rényi mutual information and joint entropy on batch covariance/Gram matrices to show that Barlow Twins and spectral contrastive losses implicitly maximize these quantities; extends the view to MAE, where they reduce to entropy, and adds a TCR-based entropy regularizer (M-MAE) that subsumes U-MAE as a second-order approximation.
Key results
- M-MAE reaches 62.4% linear probing on ImageNet with ViT-Base vs 58.5% for U-MAE and 55.4% for MAE.
- M-MAE with ViT-Large reaches 84.3% fine-tuning accuracy vs 83.3% for MAE and 83.2% for U-MAE.
- On CIFAR-100 (1000 epochs, ViT-Base), M-MAE gets 60.9% linear probe@1 vs 59.7% for U-MAE and 59.2% for MAE, but MAE fine-tunes slightly better (84.5 vs 83.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the TCR entropy regularizer to MAE, yielding M-MAE | ViT-Base on ImageNet-1K · ImageNet-1K linear probing | accuracy: 58.5 -> 62.4 | Helped |
| add the TCR entropy regularizer to MAE, yielding M-MAE | ViT-Large on ImageNet-1K · ImageNet-1K linear probing | accuracy: 65.8 -> 66.0 | Helped |
| add the TCR entropy regularizer to MAE, yielding M-MAE | ViT-Base on ImageNet-1K · ImageNet-1K fine-tuning | accuracy: 83.0 -> 83.1 | Helped |
| add the TCR entropy regularizer to MAE, yielding M-MAE | ViT-Large on ImageNet-1K · ImageNet-1K fine-tuning | accuracy: 83.2 -> 84.3 | Helped |
| use M-MAE instead of U-MAE | ViT-Base on CIFAR-100 · CIFAR-100 linear probe@1 | accuracy: 59.7 -> 60.9 | Helped |
| use M-MAE instead of U-MAE | ViT-Base on CIFAR-100 · CIFAR-100 linear probe@5 | accuracy: 86.8 -> 88.1 | Helped |
| use M-MAE instead of U-MAE | ViT-Base on CIFAR-100 · CIFAR-100 finetune@1 | accuracy: 84.1 -> 83.8 | Hurt |
| use M-MAE instead of U-MAE | ViT-Base on CIFAR-100 · CIFAR-100 finetune@5 | accuracy: 96.5 -> 97.0 | Helped |
| vary TCR coefficient μ from 1 to 0.1 (ablation) | M-MAE ViT-Base · ImageNet-1K linear probing | accuracy: 62.40 -> 58.61 | Hurt |
| vary TCR coefficient μ from 1 to 0.5 (ablation) | M-MAE ViT-Base · ImageNet-1K linear probing | accuracy: 62.40 -> 59.38 | Hurt |
| vary TCR coefficient μ from 1 to 0.75 (ablation) | M-MAE ViT-Base · ImageNet-1K linear probing | accuracy: 62.40 -> 59.87 | Hurt |
| vary TCR coefficient μ from 1 to 1.25 (ablation) | M-MAE ViT-Base · ImageNet-1K linear probing | accuracy: 62.40 -> 59.54 | Hurt |
Surprising findings
- On CIFAR-100 M-MAE's fine-tuning@1 (83.8%) is below MAE's (84.5%), so gains are not uniform across metrics.
- Linear probing accuracy for ViT-Base drops sharply as the TCR coefficient mu departs from 1 (e.g., 50.46% at mu=3 vs 62.40% at mu=1).
Limitations
- Theoretical bounds are proven mainly for alpha=2, and for general alpha the results only show maximization at zero loss.
- Hyperparameters for M-MAE on CIFAR-100 were not extensively searched, per the authors.
- Gains on fine-tuning are small (about 1% at ViT-Large) and mixed on CIFAR-100.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100
Benchmarks: ImageNet-1K
Metrics: linear probing accuracy fine-tuning accuracy KNN accuracy effective rank
Baselines: MAE U-MAE SimCLR BYOL
Builds on: MAE U-MAE Barlow Twins spectral contrastive loss matrix-based Rényi entropy
Closest papers by meaning
- Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · similarity 0.87)
- Matrix Information Theory for Self-Supervised Learning (ICML 2024 · similarity 0.87)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.85)
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.84)
- Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.83)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.