Research map / Vision-language and multimodal / Cross-modal learning
Multi-Modal Contrastive Masked Autoencoders: A Two-Stage Progressive Pre-training Approach for RGBD Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage progressive pre-training for ViTs on RGB-D data: patch-level RGB-depth contrastive learning, then masked depth autoencoding with a noise-prediction denoising loss and feature distillation from stage one.
Problem
Self-supervised pre-training of vision transformers for dense prediction using RGB-D data; existing methods either need segmentation labels, use only one objective, or fail to combine contrastive and masked modeling effectively on RGB-D data.
Approach
Combine patch-level cross-modal InfoNCE contrastive learning (stage 1) with a second stage that couples masked depth reconstruction, a diffusion-inspired noise-prediction denoising loss conditioned on noise level, and smooth-L1 feature distillation from the stage-1 encoder.
Key results
- Achieves 67.5 mIoU on ScanNet 2D semantic segmentation vs 66.2 for Mask3D and 65.1 for MultiMAE (ViT-B).
- Reaches 87.1 delta1 on NYUv2 depth estimation vs 85.4 for Mask3D and 85.6 for CroCo.
- Achieves 48.7 mIoU on SUN RGB-D semantic segmentation vs 47.4 for Mask3D; 23.7 AP on ScanNet instance segmentation vs 22.8 for Mask3D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two-stage progressive pre-training with contrastive learning, masked depth reconstruction, denoising, and feature distillation | ViT-B · ScanNet 2D semantic segmentation | mIoU: 66.2 -> 67.5 | Helped |
| Use two-stage progressive pre-training with contrastive learning, masked depth reconstruction, denoising, and feature distillation | ViT-B · SUN RGB-D 2D semantic segmentation | mIoU: 47.4 -> 48.7 | Helped |
| Use two-stage progressive pre-training with contrastive learning, masked depth reconstruction, denoising, and feature distillation | ViT-B · NYUv2 depth estimation | δ1: 85.6 -> 87.1 | Helped |
| Use two-stage progressive pre-training | Semantic segmentation model · SUN RGB-D semantic segmentation | accs: 64.3 -> 65.4 | Helped |
| Use two-stage progressive pre-training | ScanNet 2D instance segmentation model · ScanNet 2D instance segmentation | AP: 22.8 -> 23.7 | Helped |
| Add input noise only (ablation) | Denoising stage of the proposed method · ScanNet 2D semantic segmentation | mIoU: 66.5 -> 66.9 | Helped |
| Add the full denoising method, including noise prediction and decoder positional encoding (ablation) | Denoising stage of the proposed method · ScanNet 2D semantic segmentation | mIoU: 66.9 -> 67.5 | Helped |
| Add depth reconstruction to contrastive pre-training (ablation) | Proposed pre-training method · ScanNet 2D semantic segmentation | mIoU: 63.4 -> 66.3 | Helped |
| Add feature distillation (ablation) | Contrastive + reconstruction + denoising pre-training · ScanNet 2D semantic segmentation | mIoU: 67.0 -> 67.5 | Helped |
| Use higher masking ratios for both RGB and depth (ablation) | Proposed pre-training method · ScanNet 2D semantic segmentation | mIoU: 65.4 -> 67.5 | Helped |
| Add RGB reconstruction to depth reconstruction (ablation) | Second-stage masked autoencoding · ScanNet 2D semantic segmentation | mIoU: 67.5 -> 66.9 | Hurt |
| Use the proposed pre-training method instead of MAE | ViT-L · ScanNet 2D semantic segmentation | mIoU: 68.2 -> 70.8 | Helped |
Surprising findings
- Reconstructing RGB in addition to depth in stage 2 does not improve performance (66.9 vs 67.5 mIoU).
- Contrastive learning alone in stage 1 already reaches 63.4 mIoU, and the ablation shows combining contrastive and single-objective methods in one framework performed poorly according to the authors' experiments.
Limitations
- Depth maps for ImageNet are extracted with an off-the-shelf depth estimation model rather than real sensor depth.
- Pixel-level contrastive learning and pixel-to-point alignment are left for future work.
- Stage-2 reconstruction uses depth only as the target, and RGB reconstruction did not help.
Experimental setup
Datasets: ImageNet ScanNet SUN RGB-D NYUv2
Benchmarks: ScanNet 2D semantic segmentation SUN RGB-D semantic segmentation NYUv2 depth estimation ScanNet 2D instance segmentation
Metrics: mIoU accuracy delta1 AP
Baselines: MAE MultiMAE Mask3D Pri3D CoMAE CroCo DINO MoCov2
Builds on: MAE MultiMAE Mask3D InfoNCE denoising diffusion models
Closest papers by meaning
- R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.87)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.87)
- Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget (AAAI 2024 · similarity 0.86)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.86)
- DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation (ICLR 2024 · similarity 0.86)
- Towards Compact 3D Representations via Point Feature Enhancement Masked Autoencoders (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.