Research map / Vision-language and multimodal / CLIP and zero-shot
The Effectiveness of MAE Pre-Pretraining for Billion-Scale Pretraining
Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollar, Christoph Feichtenhofer, Ross Girshick, Rohit Girdhar, Ishan Misra
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MAE pre-pretraining before weakly supervised pretraining (ablation) | WSP on IG-3B · ImageNet-1k linear probe | top-1 accuracy | Improved |
| add MAE pre-pretraining (2B vs 6.5B without) (ablation) | WSP · ImageNet-1k linear probe | top-1 accuracy | Improved |
| increase pre-pretraining MAE epochs beyond 1 (ablation) | MAE->WSP · IG-3B pretraining | transfer accuracy | No clear effect |
| add MAE pre-pretraining under fixed training FLOPs (ablation) | WSP · IG-3B pretraining | transfer performance: up to 2x more efficient | Improved |
| pre-pretrain MAE on ImageNet-1k instead of IG-3B (ablation) | MAE->WSP ViT-H · ImageNet-1k finetune | top-1 accuracy: 89.3 -> 89.4 | No clear effect |
| add MAE pre-pretraining to WSP for object detection scaling (ablation) | WSP ViT · LVIS/COCO detection | AP | Improved |
| add MAE pre-pretraining on ImageNet-21k pretraining (ablation) | WSP ViT-L · IN21k pretraining | transfer accuracy | Improved |
| scale MAE pretraining data from IN1k to IG-3B (ablation) | MAE · downstream tasks | accuracy | Improved |
| MAE pre-pretraining (MAWS ViT-H) vs MAE on IN1k for COCO detection | MAWS ViT-H · COCO val2017 | AP box: 58.7 -> 57.7 | Hurt |
| add MAE pre-pretraining to SWAG-style WSP ViT-H on LVIS | WSP ViT-H · LVIS v1 val | AP box: 47.1 -> 50.8 | Improved |
| add relative position embeddings in video finetuning (ablation) | MAWS ViT-L · Kinetics-400 | top-1 accuracy: +0.6 | Improved |
| scale WSP training hyperparameters choices (loss, head, augmentation, lr) (ablation) | WSP · IG-3B 1 epoch | accuracy | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 0 of 1 on most shared results.
- CoCa-2B — COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering (AAAI 2023) · better on 0 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Revisiting the Transferability of Supervised Pretraining: An MLP Perspective (CVPR 2022 · similarity 0.89)
- Self-Supervised Pretraining of 3D Features on Any Point-Cloud (ICCV 2021 · similarity 0.89)
- ImageNet-21K Pretraining for the Masses (NeurIPS 2021 · similarity 0.89)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.88)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.87)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
