Research map / Label-efficient and robust learning / Self-supervised and contrastive
Learning Mask Invariant Mutual Information for Masked Image Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MI-MAE, which adds mutual-information maximization across orthogonal masks and minimization (CLUB-style upper bound) of input-latent MI to masked autoencoders, motivated by an information bottleneck analysis.
Problem
Understanding why masked autoencoders work is incomplete, and existing analyses are implicit (contrastive or feature-based); the paper seeks an information-theoretic account and a method to improve MAE latent features.
Approach
Views MAE latent features through the information bottleneck principle and optimizes them with InfoNCE-based MI maximization across N orthogonal masks of the same image plus a CLUB-style variational MI minimization between latent features and masked inputs, added to the reconstruction loss.
Key results
- MI-MAE pre-trained 400 epochs reaches 84.1% ImageNet-1K fine-tuning accuracy vs. 83.6% for the 1600-epoch MAE, with 69.3% linear probing (+1.3).
- On COCO, MI-MAE reports 52.0 AP^box and 46.1 AP^mask vs. 51.2 and 45.5 for MAE; on ADE20K, 49.3 mIoU vs. 48.1.
- Ablation on ImageNet-1K (50-epoch pretraining): full loss gives 82.8% vs. 82.2% for the reconstruction-only baseline; independent masks instead of orthogonal drop accuracy by 0.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add mutual-information optimization to MAE | MAE with ViT-B · ImageNet-1K | FT accuracy: 83.6 -> 84.1 | Helped |
| Add mutual-information optimization to MAE | MAE with ViT-B · COCO | APbox: 51.2 -> 52.0 | Helped |
| Add mutual-information optimization to MAE | MAE with ViT-B · COCO | APmask: 45.5 -> 46.1 | Helped |
| Add mutual-information optimization to MAE | MAE with ViT-B · ADE20K | mIoU: 48.1 -> 49.3 | Helped |
| Add mutual-information optimization to MAE | MAE · ImageNet-C | mCE: 51.7 -> 49.5 | Helped |
| Add max-MI loss (ablation) | MAE with reconstruction loss · ImageNet-1K | accuracy: 82.2 -> 82.5 | Helped |
| Remove reconstruction loss and use max-MI loss only (ablation) | MI-MAE · ImageNet-1K | accuracy | Hurt |
| Add min-MI loss without training the approximation network (ablation) | MAE + max-MI · ImageNet-1K | accuracy: 82.5 -> 82.4 | Hurt |
| Add approximation loss to train the approximation network (ablation) | MAE + min-MI · ImageNet-1K | accuracy: 82.4 -> 82.4 | Helped |
| Add the complete mutual-information losses (ablation) | MAE with reconstruction loss · ImageNet-1K | accuracy: 82.5 -> 82.8 | Helped |
| Set λ3 to 10 rather than 1 (ablation) | MI-MAE · ImageNet-1K | accuracy: 82.6 -> 82.8 | Helped |
| Replace orthogonal masks with independent masks (ablation) | MI-MAE · ImageNet-1K | accuracy: 82.8 -> 82.6 | Hurt |
Surprising findings
- Using only the mutual-information maximization loss without reconstruction loss causes training collapse.
- The information-bottleneck-motivated losses use a 0.5 reconstruction-loss threshold for activating the MI terms, with the MAE loss used alone before that.
Limitations
- Theoretical derivations rely on strong assumptions (e.g., Assumption on reconstruction loss threshold and sub-optimal latent features) and approximations, which are loosely justified.
- Gains on fine-tuning are modest (about 0.5 points over 1600-epoch MAE) and most comparisons use the 800/1600-epoch MAE numbers from other papers.
- Results are reported on ViT-B and ViT-S; experiments on larger models are not shown.
Experimental setup
Datasets: ImageNet-1K COCO ADE20K
Benchmarks: ImageNet-1K COCO ADE20K ImageNet-A ImageNet-C
Metrics: top-1 accuracy linear probing accuracy low-shot fine-tuning accuracy box AP mask AP mIoU mCE
Baselines: MAE SimMIM PixMIM MFF SemMAE C-MAE MoCo v3 BEiT DINO
Builds on: masked autoencoders information bottleneck InfoNCE CLUB MINE
Closest papers by meaning
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.92)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.90)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.89)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.89)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.89)
- R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.