Research map / Detection and segmentation / 3D detection and driving
SAM-Guided Masked Token Prediction for 3D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage SAM-guided masked token prediction framework for 3D scene understanding that replaces KNN tokenization with SAM-mask tokenization, adds group-balanced re-weighting, and distills from DINOv2 into a plain transformer 3D encoder.
Problem
Knowledge distillation from 2D foundation models to 3D transformers suffers from misalignment between KNN-based point tokens and SAM regions, and from long-tail distributions in 3D datasets.
Approach
Tokenize point clouds by projecting them to images and grouping points by SAM masks instead of KNN, weight region-level distillation loss by pseudo-label cluster groups, and use a frozen stage-1 teacher to supervise a masked student with instance-level and token-level feature prediction.
Key results
- On ScanNetV2 object detection with the 3DETR baseline, the method reaches 68.2 AP25 and 48.4 AP50, versus 65.3 and 44.2 for Bridge3D.
- On S3DIS semantic segmentation, mIoU rises to 71.8 from 70.2 for Bridge3D; on ScanNetV2 segmentation it reaches 75.4 mIoU versus 73.9.
- Ablation shows SAM-guided tokenization gives the largest single gain (ScanNetV2 AP50 from 44.3 to 47.0 when added to dense distillation and masked token prediction).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full SAM-guided masked token prediction method instead of Bridge3D | 3DETR · ScanNetV2 | AP25: 65.3 -> 68.2 (+2.9) | Helped |
| Use the full method instead of Bridge3D | GroupFree3D · ScanNetV2 | AP50: 51.9 -> 55.7(+3.8) | Helped |
| Add dense distillation (ablation) | Plain Transformer · ScanNetV2 | AP25: 61.1 -> 62.4 | Helped |
| Add masked token prediction to dense distillation (ablation) | Plain Transformer · ScanNetV2 | AP25: 62.4 -> 64.5 | Helped |
| Add balanced re-weighting to dense distillation and masked token prediction (ablation) | Plain Transformer · ScanNetV2 | AP25: 64.5 -> 66.0 | Helped |
| Add SAM-guided tokenization to dense distillation and masked token prediction (ablation) | Plain Transformer · ScanNetV2 | AP25: 64.5 -> 67.1 | Helped |
| Use the two-stage teacher-student framework rather than adding masked token prediction in Stage 1 (ablation) | Proposed method · ScanNetV2 | AP25: 66.0 -> 68.2 | Helped |
Limitations
- Relies on offline SAM masks and 2D image features, requiring paired images with point clouds.
- Group-balanced weighting uses pseudo-labels from K-means-style clustering of foundation-model features rather than true class labels.
- Comparisons with methods using different backbones were adapted by the authors to transformers in a rebuttal-style experiment.
Experimental setup
Datasets: ScanNetV2 SUN RGB-D S3DIS
Benchmarks: SUN RGB-D ScanNetV2 S3DIS
Metrics: AP25 AP50 mIoU mAcc
Baselines: Bridge3D 3DETR GroupFree3D Point-MAE PiMAE CLIP2Scene Seal PPT DepthContrast PointContrast CAGroup3D VDETR
Builds on: Bridge3D Point-MAE 3DETR GroupFree3D MaskCLIP
Closest papers by meaning
- Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understanding (CVPR 2025 · similarity 0.87)
- XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation (NeurIPS 2024 · similarity 0.87)
- Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.86)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.86)
- Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · similarity 0.86)
- SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.