Research map / Label-efficient and robust learning / Recognition and long-tail
Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking
Peng Gao, Ziyi Lin, Renrui Zhang, Rongyao Fang, Hongyang Li, Hongsheng Li, Yu Qiao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mimic loss on visible encoder tokens (ablation) | MAE (200 ep) · ImageNet-1K fine-tuning | top-1 accuracy (%): 83.0 -> 84.7 | Improved |
| add focused mimicking (salient tokens) (ablation) | MR-MAE · ImageNet-1K fine-tuning | top-1 accuracy (%): 84.7 -> 84.9 | Improved |
| add multi-layer fusion (ablation) | MR-MAE · ImageNet-1K fine-tuning | top-1 accuracy (%): 84.9 -> 85.0 | Improved |
| add masked convolution stages (ablation) | MR-MAE · ImageNet-1K fine-tuning | top-1 accuracy (%): 85.0 -> 85.5 | Improved |
| add masked convolution stages (ablation) | MR-MAE · COCO | AP box: 51.6 -> 53.0 | Improved |
| apply reconstruction to both tokens instead of mimic before reconstruct (ablation) | MR-MAE · ImageNet-1K fine-tuning | top-1 accuracy (%): 85.0 -> 83.3 | Hurt |
| add low-level reconstruction to high-level mimicking (ablation) | MR-MAE · COCO | AP box: 50.9 -> 51.6 | Improved |
| use CLIP teacher instead of DINO (ablation) | MR-MAE · ImageNet-1K fine-tuning | top-1 accuracy (%): 84.0 -> 85.0 | Improved |
| mimic CLIP and DINO jointly instead of CLIP alone (ablation) | MR-MAE · ImageNet-1K fine-tuning | top-1 accuracy (%): 85.0 -> 83.8 | Hurt |
| ensemble separately trained CLIP and DINO models instead of joint (ablation) | MR-MAE · ImageNet-1K fine-tuning | top-1 accuracy (%): 83.8 -> 85.5 | Improved |
| pre-train 400 instead of 200 epochs (ablation) | MR-MAE · ImageNet-1K fine-tuning | top-1 accuracy (%): 85.5 -> 85.8 | Improved |
| pre-train 800 instead of 400 epochs (ablation) | MR-MAE · ImageNet-1K fine-tuning | top-1 accuracy (%): 85.8 -> 85.8 | No clear effect |
Closest papers by meaning
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.89)
- Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.88)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.88)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.88)
- R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.87)
- Improved Baselines with Representation Autoencoders (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
