Research map / Label-efficient and robust learning / Self-supervised and contrastive
Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Masked Scene Modeling, a hierarchical bottom-up masked self-supervised method for 3D indoor scenes that reconstructs teacher deep features at every UNet level, plus a multi-resolution linear-probing/NN evaluation protocol.
Problem
Self-supervised 3D scene models are mainly used as weight initialization, and their off-the-shelf feature quality is poorly measured and lags far behind supervised models; hierarchical UNet-style models also lose information in the decoder.
Approach
Mask sparse voxels in the encoder input only, reinsert masked tokens at the decoder, and reconstruct EMA-teacher deep features at each hierarchy level with cross-view prediction; evaluate with tri-linearly upsampled concatenated decoder features rather than the last layer alone.
Key results
- Linear probing on ScanNet semantic segmentation reaches 68.7 mIoU versus 37.3 for the best prior self-supervised baseline (MSC, same architecture) and 77.0 for supervised HUNet.
- NN protocol on ScanNet reaches 65.7 mIoU, outperforming the best prior self-supervised baseline by more than 30 points.
- On ScanRefer overall accuracy, the method reaches 55.7% versus 51.0% for supervised HUNet and 45.0% for MSC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · ScanNet semantic segmentation | mIoU: +25 | Helped |
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · ScanNet200 semantic segmentation | mIoU: +11 | Helped |
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · S3DIS semantic segmentation | mIoU: +6 | Helped |
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · ScanNet semantic segmentation, linear probing | mIoU: +10 | Helped |
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · ScanNet200 semantic segmentation, linear probing | mIoU: +6 | Helped |
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · S3DIS semantic segmentation, linear probing | mIoU: +2 | Helped |
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · ScanNet instance segmentation | mAP@50: +30 | Helped |
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · ScanNet200 instance segmentation | mAP@50: +7 | Helped |
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · S3DIS instance segmentation | mAP@50: +6 | Helped |
| Use Masked Scene Modeling instead of existing 3D self-supervised objectives | HUNet · ScanRefer | accuracy: +10, +7, and +10 | Helped |
| Train with Masked Scene Modeling rather than MSC | HUNet · ScanNet semantic segmentation with 10% of training scenes | mIoU | Helped |
| Use hierarchical features from all decoder levels rather than only the last layer | Self-supervised 3D models · ScanNet semantic segmentation, linear probing | mIoU | Helped |
Surprising findings
- Using only last-layer decoder features severely underestimates existing self-supervised models; hierarchical multi-level features raise their linear probing scores substantially in the pilot study.
- Training HUNet from scratch on 3D visual grounding was unstable and failed to converge.
Limitations
- Pre-training uses only the relatively small ScanNet dataset.
- Evaluation of 3D visual grounding uses only the object discriminator with ground-truth boxes.
Experimental setup
Datasets: ScanNet ScanNet200 S3DIS ScanRefer
Benchmarks: ScanNet ScanNet200 S3DIS ScanRefer
Metrics: mIoU mAP@50 accuracy
Baselines: CSC MSC MM3D OESSL SR-UNet supervised DINOv2 LSeg CLIP Stable Diffusion features
Builds on: masked image modeling DINOv2 PointGroup Point Transformer V3 ScanNet
Closest papers by meaning
- MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling With Informative-Preserved Reconstruction and Self-Distilled Consistency (CVPR 2023 · similarity 0.88)
- DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features (NeurIPS 2024 · similarity 0.88)
- SAM-Guided Masked Token Prediction for 3D Scene Understanding (NeurIPS 2024 · similarity 0.87)
- Sonata: Self-Supervised Learning of Reliable Point Representations (CVPR 2025 · similarity 0.86)
- Asymmetric Dual Self-Distillation for 3D Self-Supervised Representation Learning (NeurIPS 2025 · similarity 0.86)
- Probing the Mid-level Vision Capabilities of Self-Supervised Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.