Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-guided Masked Autoencoders (SMA), a fully domain-agnostic masked modeling method that learns to sample masks from the model's own first-layer attention maps, evaluated on protein, chemistry, particle physics, and some image/text benchmarks.
Problem
Masked modeling is a promising domain-agnostic self-supervised framework, but its mask sampling and tokenization remain domain-specific, hindering extension to new modalities such as scientific data.
Approach
Sample masks from the masked prediction model's own attention maps: sum attention rows of a random subset of queries, mask the top-k tokens (approximated for parallelism), and train end-to-end with a reconstruction loss, using only character/value-level inputs with no tokenizer or augmentations.
Key results
- On TAPE, SMA reaches 0.23 Remote Homology, 0.68 Fluorescence, 0.80 Stability, beating ProteinBERT (0.22/0.66/0.76).
- On HIGGS small fine-tuning, guided masking reaches 74.8% accuracy vs 73.0% for the best prior supervised model (Transformer-PLR) and 74.0% for random masking.
- On GLUE, SMA averages 76.7 vs 75.6 for word masking with the same Perceiver IO architecture.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use attention-guided masks instead of random masking | Perceiver IO pretraining · TAPE Remote Homology | accuracy: 0.22 -> 0.23 | Helped |
| Use attention-guided masks instead of random masking | Perceiver IO pretraining · TAPE Fluorescence | Spearman correlation: 0.67 -> 0.68 | Helped |
| Use attention-guided masks instead of random masking | Perceiver IO pretraining · TAPE Stability | Spearman correlation: 0.76 -> 0.80 | Helped |
| Use SMA instead of Uni-Mol-20M | MoleculeNet · BBBP | ROC: 0.729 -> 0.750 | Helped |
| Use SMA instead of Uni-Mol-20M | MoleculeNet · BACE | ROC: 0.857 -> 0.843 | Hurt |
| Use SMA instead of random masking | Transformer pretraining · HIGGS test, fine-tune 1k, 10k, and 100k | accuracy (%): 68.07 -> 69.47; 72.57 -> 74.04; 76.54 -> 77.88 | Helped |
| Use SMA instead of word-level masking | Perceiver IO pretraining · GLUE evaluation | Avg: 75.6 -> 76.7 | Helped |
| Use SMA instead of random masking | Perceiver IO image pretraining · ImageNet-100 evaluation, w/ augmentation | Top-1 accuracy: 85.6 -> 86.1 | Helped |
| Replace sequential top-k mask selection with the parallel approximation | SMA masking · Image domain | steps/second: 1.13 -> 3.58 | Helped |
| Replace sequential top-k mask selection with the parallel approximation | SMA masking · Across domains | performance | No clear effect |
| Use SMA instead of random masking | Transformer image pretraining · ImageNet-100 evaluation, w/o augmentation | Top-1 accuracy: 77.6 -> 78.5 | Helped |
| Use SMA instead of Uni-Mol-20M | MoleculeNet · FreeSolv | RMSE: 1.48 -> 1.09 | Helped |
Surprising findings
- Random masking outperforms patch masking on ImageNet-100 when strong augmentations are used during fine-tuning (85.6 vs 84.3), but performs worse without augmentation.
Limitations
- Gains on GLUE are modest and the reproduced word-masking baseline uses a smaller model and pretraining set than the original.
- Learned masks may have low effective masking ratio early in training, causing instability from overlapping query attention.
- Image experiments still use a resize to 192x192, which involves some domain knowledge.
Experimental setup
Datasets: Pfam TAPE guacamol MoleculeNet HIGGS English Wikipedia BooksCorpus GLUE ImageNet-100
Benchmarks: TAPE MoleculeNet HIGGS GLUE ImageNet-100 year prediction
Metrics: accuracy Spearman correlation ROC AUC RMSE F1
Baselines: ProteinBERT ChemBERTa-2 Uni-Mol TabNet Perceiver IO word masking Random Masking Patch masking FT-Transformer XGBoost CatBoost
Builds on: Perceiver IO masked autoencoders TAPE benchmark setup Transformer
Closest papers by meaning
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.89)
- Self-supervised Masked Graph Autoencoder via Structure-aware Curriculum (ICML 2025 · similarity 0.88)
- Exploring Target Representations for Masked Autoencoders (ICLR 2024 · similarity 0.88)
- Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology (CVPR 2024 · similarity 0.87)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.86)
- Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understanding (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.