Research map / Vision-language and multimodal / Human and embodied modeling
Understanding and Enhancing Mask-Based Pretraining towards Universal Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes mask-based pretraining via test risk of ridge-less linear regression and proposes R²MAE, which samples mask ratios uniformly from a range, improving vision, language, DNA, and single-cell models.
Problem
Mask-based pretraining works across domains with very different optimal masking ratios, and its underlying mechanism and the effect of model size on the optimal mask ratio are not well understood.
Approach
Model feature-level mask autoencoding as minimum-norm (ridge-less) linear regression with random masks, deriving asymptotic test risk that reproduces the sweet-spot masking curves; guided by this, sample the masking ratio uniformly from a range per training example (R²MAE) to learn multi-scale features.
Key results
- R²MAE reaches 82.00 top-1 on ImageNet fine-tuning with ViT-base MAE vs 81.97 for default MR 0.75 (marginal gain).
- On GLUE with RoBERTa-base, R²MAE attains the best average rank (1.25) tied with RoBERTa-medium, e.g., 81.9 MNLI and 91.9 SST-2.
- On single-cell SEA-AD benchmarks, R²MAE achieves the best average score 0.697 and rank 2.17 among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed mask ratio 0.75 with a uniformly sampled ratio from 0.6 to 0.9 | ViT-base MAE · ImageNet classification | Top1 Acc.: 81.97 -> 82.00 | Helped |
| replace fixed mask ratio 0.75 with a uniformly sampled ratio from 0.6 to 0.9 | ViT-base MAE · ImageNet classification | Top5 Acc.: 96.02 -> 96.05 | Helped |
| sample mask ratio uniformly from 0.15 to 0.4 instead of using default MLM | RoBERTa-Base · GLUE SST-2 | accuracy: 91.7 -> 91.9 | Helped |
| sample mask ratio uniformly from 0.15 to 0.4 instead of using default MLM | RoBERTa-Medium · GLUE QNLI | accuracy: 87.1 -> 86.7 | Hurt |
| sample masking ratio from a predefined range instead of fixed MLM | GPN-MSA · OMIM regulatory | AUPRC: 0.127 -> 0.148 | Helped |
| sample masking ratio from a predefined range instead of fixed MAE mask ratio 25% | single-cell MLP · SEA-AD brain single-cell | Avg performance score: 0.667 -> 0.697 | Helped |
| add Dynamic MR to R²MAE (ablation) | GPN-MSA · OMIM regulatory | AUPRC: 0.148 -> 0.138 | Hurt |
| add CL to R²MAE (ablation) | GPN-MSA · OMIM subset class ncRNA | AUPRC: 0.136 -> 0.192 | Helped |
| add CL to R²MAE (ablation) | GPN-MSA · OMIM subset class 3'UTR | AUPRC: 0.050 -> 0.058 | Helped |
| add CL to R²MAE (ablation) | single-cell MLP · SEA-AD brain single-cell | Avg performance score: 0.697 -> 0.683 | Hurt |
| add Dynamic MR to R²MAE (ablation) | single-cell MLP · SEA-AD brain single-cell | Avg performance score: 0.697 -> 0.665 | Hurt |
| add fixed curriculum CL (k=0) to R²MAE (ablation) | single-cell MLP · SEA-AD brain single-cell | Avg performance score: 0.697 -> 0.662 | Hurt |
Surprising findings
- Fixed-ratio MAE/MLM with plain high or low masking can underperform, while the marginal ImageNet gain of R²MAE over default MAE is only 0.03 top-1.
- A fixed curriculum with k=0 (learned loss-weighting) caused learning degeneration, while R²MAE-CL resolved it.
Limitations
- Theoretical characterization does not extend to R²MAE itself; explicit test risk for the random-ratio scheme remains open.
- Gains on ImageNet are marginal (e.g., 82.00 vs 81.97), and R²MAE's pretraining loss fluctuates in ViT-MAE.
- Vision experiments trained for only 150 epochs versus the default 800, so absolute accuracy is suboptimal.
Experimental setup
Datasets: ImageNet FineWeb GLUE MNIST CelebA Clinvar OMIM SEA-AD
Benchmarks: ImageNet GLUE Clinvar OMIM SEA-AD
Metrics: top-1 accuracy top-5 accuracy accuracy F1 AUROC AUPRC Spearman correlation normalized test risk
Baselines: MAE default 0.75 mask ratio MDLM Dynamic MR CL-MAE MLM default 0.15 GPN-MSA scGPT scVI
Builds on: MAE BERT RoBERTa MDLM Dynamic MR
Closest papers by meaning
- VideoMAE V2: Scaling Video Masked Autoencoders With Dual Masking (CVPR 2023 · similarity 0.89)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.89)
- Video Language Model Pretraining with Spatio-temporal Masking (CVPR 2025 · similarity 0.88)
- Asymmetric Masked Distillation for Pre-Training Small Foundation Models (CVPR 2024 · similarity 0.88)
- Efficient Vision-Language Pre-training by Cluster Masking (CVPR 2024 · similarity 0.88)
- R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.