Research map / Label-efficient and robust learning / Self-supervised and contrastive
The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CMT-MAE, a masked autoencoder that linearly aggregates teacher (CLIP) and momentum-student attention maps to guide masking and reconstructs both teacher and student features as collaborative targets.
Problem
Prior teacher-guided masked image modeling uses fixed teachers (e.g., CLIP) for masks and targets and ignores feedback from the self-training student model during pre-training.
Approach
Collaborative masking uses a linear mix (collaborative ratio alpha) of teacher attention maps and momentum-student attention maps to choose masked patches, and the decoder reconstructs both teacher and student features with an alpha-weighted loss.
Key results
- ViT-B/16 fine-tuning on ImageNet-1K improves from 83.6% (MAE) to 85.7%, and linear probing from 68.0% to 79.8%.
- ADE20K mIoU rises from 48.1 (MAE) to 52.9 with ViT-B/16.
- DAVIS 2017 (J&F)m rises from 51.0 (MAE) to 57.6 with ViT-B/16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CMT-MAE instead of MAE pre-training | ViT-B/16 · ImageNet-1K fine-tuning | top-1 accuracy: 83.6% to 85.7% | Helped |
| use CMT-MAE instead of MAE pre-training | ViT-B/16 · ImageNet-1K linear probing | top-1 accuracy: 68.0% to 79.8% | Helped |
| use CMT-MAE instead of MAE pre-training | ViT-B/16 · ADE20K semantic segmentation | mIoU: 48.1 → 52.9 | Helped |
| use CMT-MAE instead of MAE pre-training | ViT-B/16 · DAVIS 2017 video segmentation | (J & F)m: 51.0 → 57.6 | Helped |
| use CMT-MAE instead of MAE pre-training | ViT-B/16 · COCO object detection | APbox: 50.3 → 52.8 | Helped |
| use CMT-MAE instead of MAE pre-training | ViT-B/16 · COCO instance segmentation | APmask: 44.9 → 45.7 | Helped |
| add collaborative masking to the baseline (ablation) | CMT-MAE baseline · ImageNet-1K linear probing | top-1 accuracy: 68.0 to 73.5 | Helped |
| add collaborative targets to the baseline (ablation) | CMT-MAE baseline · ImageNet-1K linear probing | top-1 accuracy: 68.0 to 74.2 | Helped |
| add collaborative masking and collaborative targets (ablation) | CMT-MAE baseline · ImageNet-1K linear probing | top-1 accuracy: 68.0 to 79.8 | Helped |
| increase collaborative ratio from 0% to 30% (ablation) | CMT-MAE · ImageNet-1K linear probing | top-1 accuracy: 77.1 to 79.8 | Helped |
| increase collaborative ratio from 30% to 90% (ablation) | CMT-MAE · ImageNet-1K linear probing | top-1 accuracy: 79.8 to 78.3 | Hurt |
Surprising findings
- Increasing the collaborative ratio beyond 30% (up to 100%) does not keep improving results, suggesting a teacher-student trade-off.
Limitations
- Gains over the MAE baseline are reported mainly with the paper's own settings; the paper gives no compute or hardware details.
- Collaborative ratio alpha must be tuned, with best results at 30% and degradation at higher values.
Experimental setup
Datasets: ImageNet-1K MS-COCO ADE20K DAVIS 2017
Benchmarks: ImageNet-1K MS-COCO ADE20K DAVIS 2017
Metrics: top-1 accuracy AP box AP mask mIoU J&F mean J mean F mean
Baselines: MAE MVP AttMask SemMAE iBoT DINO MoCo v3 BEiT MaskFeat SimMIM I-JEPA data2vec
Builds on: masked autoencoder (MAE) MVP AttMask iBoT DINO
Closest papers by meaning
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.90)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.89)
- Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · similarity 0.89)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.89)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.89)
- VideoMAC: Video Masked Autoencoders Meet ConvNets (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.