Research map / Detection and segmentation / Open-set and few-shot detection
R-MAE: Regions Meet Masked Autoencoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R-MAE, which adds a masked region autoencoding branch (regions as queries, length variant) to MAE so ViT pixel encoders learn region-aware representations, pre-trained mainly on COCO with FH regions.
Problem
MAE's pixel-reconstruction pretext lags in object-level understanding; the paper asks whether regions (groups of pixels) can serve as a discrete visual analogue of words for self-supervised image representation learning, especially for detection and segmentation.
Approach
Masked region autoencoding reconstructs binary region maps from visible region patches; a permutation-equivariant "length" variant compresses each region into a single query vector that is spatially expanded via cross-attention addition, making multi-region reconstruction cheap; it is jointly trained with MAE using a pixel-to-region feed.
Key results
- R-MAE with FH regions reaches 52.3 AP^box and 46.4 AP^mask on COCO detection/instance segmentation with ImageNet pre-training (1600 epochs), vs. MAE's 50.1/44.6 on COCO-only pre-training baseline table and ahead of BEiT (49.8/44.4) at 1x FLOPs.
- RAE alone with SAM regions (50.6 AP^box, 46.8 mIoU) outperforms MAE (50.1 AP^box, 45.9 mIoU) with less compute.
- R-MAE adds only about 1% FLOPs over MAE (9.8b vs 9.7b) while improving COCO AP^box from 50.1 to 50.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FH masked-region reconstruction to MAE | R-MAE, compared with MAE · COCO val, COCO pre-training | box AP: 50.1 -> 50.6 | Helped |
| Add FH masked-region reconstruction to MAE | R-MAE, compared with MAE · ADE20K, COCO pre-training | mIoU: 45.9 -> 46.8 | Helped |
| Use SAM regions instead of FH regions (ablation) | RAE · COCO val and ADE20K | box AP: 47.2 -> 50.6 | Helped |
| Use SAM regions instead of FH regions (ablation) | RAE · ADE20K | mIoU: 42.1 -> 46.8 | Helped |
| Use panoptic ground-truth regions instead of FH regions (ablation) | RAE · ADE20K | mIoU: 42.1 -> 43.4 | Helped |
| Use panoptic ground-truth regions instead of FH regions (ablation) | RAE · COCO detection and instance segmentation | box AP: 47.2 -> 45.9 | Hurt |
| Use the batch architecture instead of channel architecture (ablation) | RAE · COCO val | box AP: 46.1 -> 47.2 | Helped |
| Use the length architecture instead of batch architecture (ablation) | RAE · COCO val | mIoU: 43.3 -> 42.1 | Hurt |
| Increase the number of sampled regions per image (ablation) | RAE · COCO detection and segmentation | performance | Helped |
| Use pixel-to-region cross-feeding instead of bidirectional cross-feeding (ablation) | R-MAE · COCO detection and segmentation and ADE20K | box AP: 50.3 -> 50.6 | Helped |
| Use pixel-to-region cross-feeding instead of region-to-pixel cross-feeding (ablation) | R-MAE · COCO detection and segmentation and ADE20K | box AP: 50.3 -> 50.6 | Helped |
| Set the pre-training learning rate to 1e-4 instead of 1.5e-4 (ablation) | MAE · COCO val | box AP: 49.9 -> 50.1 | No clear effect |
Surprising findings
- RAE alone with high-quality SAM regions outperforms full MAE on detection and segmentation.
- R-MAE trained with reconstruction of regions has much worse linear-probe ImageNet accuracy (60.6 vs 68.0) despite equal fine-tuning accuracy (83.6).
Limitations
- Best results rely on SAM regions, which initialize from MAE, are computationally expensive, and need human-in-the-loop supervision.
- Linear-probe ImageNet performance drops substantially (60.6 vs 68.0), indicating more local rather than global features.
- Regions are pre-computed and fixed rather than jointly learned with representations.
Experimental setup
Datasets: COCO train2017 COCO unlabeled ImageNet ADE20K LVIS
Benchmarks: COCO object detection COCO instance segmentation ADE20K LVIS ImageNet classification
Metrics: AP box AP mask mIoU AP rare top-1 accuracy
Baselines: MAE MoCo v3 BEiT supervised SemMAE MixedAE MultiMAE LoMaR Long-Seq MAE
Builds on: Masked Autoencoders (MAE) ViTDet DETR Felzenszwalb-Huttenlocher segmentation
Closest papers by meaning
- Self-Guided Masked Autoencoder (NeurIPS 2024 · similarity 0.90)
- Learning Mask Invariant Mutual Information for Masked Image Modeling (ICLR 2025 · similarity 0.88)
- The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning (AAAI 2025 · similarity 0.88)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.88)
- Closest Neighbors are Harmful for Lightweight Masked Auto-encoders (CVPR 2025 · similarity 0.88)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.