Research map / 3D vision / Depth and stereo
In Pursuit of Pixel Supervision for Visual Pre-training
Lihe Yang, Shang-Wen Li, Yang Li, Xinjie Lei, Dong Wang, Abdelrahman Mohamed, Saining Xie, Hengshuang Zhao, Kaiming He, Hu Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on self-curated MetaCLIP-S instead of uncurated MetaCLIP 2B (ablation) | MAE-style encoder · ADE20K | mIoU: 44.7 -> 46.8 | Improved |
| Pretrain on self-curated MetaCLIP-S instead of uncurated MetaCLIP 2B (ablation) | MAE-style encoder · NYUv2 depth | RMSE: 0.351 -> 0.321 | Improved |
| Use IN-21K instead of IN-1K as pretraining data (ablation) | MAE-style encoder · ADE20K | mIoU: 42.9 -> 44.8 | Improved |
| Increase decoder depth from 8 to 32 (ablation) | MAE on IN-21K · IN-1K k-NN | accuracy: 35.3 -> 55.8 | Improved |
| Increase decoder depth from 8 to 32 (ablation) | MAE on IN-21K · ADE20K | mIoU: 35.8 -> 40.4 | Improved |
| Make decoder even larger (768x32) (ablation) | MAE on IN-21K · dense downstream tasks | overall | Hurt |
| Mask 2x2 patch blocks instead of single patches (ablation) | MAE (512x8 decoder, 75% mask) · IN-1K k-NN | accuracy: +19.0 | Improved |
| Mask 2x2 patch blocks instead of single patches (ablation) | MAE (512x8 decoder, 75% mask) · NYUv2 depth | RMSE: 0.431 -> 0.362 | Improved |
| Increase masking block to 8x8 patches (ablation) | MAE · downstream tasks | overall | Hurt |
| Increase class tokens from 1 to 4 (ablation) | MAE · IN-1K k-NN | accuracy: 63.3 -> 75.1 | Improved |
| Increase class tokens from 1 to 4 (ablation) | MAE · dense prediction tasks | overall | No clear effect |
| Combine all three MAE updates (decoder, mask block, CLS tokens) (ablation) | MAE on curated 2B images · NYUv2 depth | RMSE: 0.392 -> 0.321 | Improved |
Closest papers by meaning
- Improving Visual Representation Learning Through Perceptual Understanding (CVPR 2023 · similarity 0.87)
- Masked Autoencoders As Spatiotemporal Learners (NeurIPS 2022 · similarity 0.87)
- Semantic Concentration for Self-Supervised Dense Representations Learning (TPAMI 2025 · similarity 0.87)
- Win-Win: Training High-Resolution Vision Transformers from Two Windows (ICLR 2024 · similarity 0.87)
- TIPS: Text-Image Pretraining with Spatial awareness (ICLR 2025 · similarity 0.87)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
