Research map / 3D vision / 3D generation
EmbodiedSAM: Online Segment Any 3D Thing in Real Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EmbodiedSAM (ESAM), an online, real-time 3D instance segmentation framework that lifts SAM 2D masks into 3D queries via geometric-aware pooling and a dual-level query decoder, and merges masks across frames with fast matrix-based similarity.
Problem
Online, real-time, fine-grained and generalizable 3D instance segmentation from streaming RGB-D video for embodied agents, where existing VFM-assisted 3D methods are offline or too slow and merge masks with hand-crafted rules.
Approach
Lift SAM-generated 2D masks into fixed-size 3D queries using geometric-aware superpoint pooling, refine them with a dual-level decoder (superpoint cross-attention, point-level mask prediction), and learn auxiliary box, contrastive and semantic embeddings so that cross-frame mask matching becomes a single matrix multiplication plus bipartite matching.
Key results
- On ScanNet200 class-agnostic segmentation, ESAM reaches 42.2 AP / 63.7 AP50 / 79.6 AP25 at 80 ms per frame excluding SAM, versus 20.2 AP for SAM3D.
- ESAM-E (FastSAM) reaches 43.4 AP on ScanNet200 at about 10 FPS, outperforming offline SAI3D (28.2 AP).
- On zero-shot transfer from ScanNet200, ESAM gets 28.8 AP on SceneNN and 14.1 AP on 3RScan, above offline SAI3D (18.6 and 8.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ESAM instead of SAM3D | 3D instance segmentation · ScanNet200 val | AP: 20.2 -> 42.2 | Helped |
| Replace SAM with FastSAM | ESAM · ScanNet200 val | AP: 42.2 -> 43.4 | Helped |
| Use ESAM instead of SAM3D | 3D instance segmentation · ScanNet200 to SceneNN | AP: 15.1 -> 28.8 | Helped |
| Use ESAM instead of SAM3D | 3D instance segmentation · ScanNet200 to 3RScan | AP: 6.2 -> 14.1 | Helped |
| Fuse FastSAM backbone image features into point clouds | ESAM-E · SceneNN | AP: 27.5 -> 33.3 | Helped |
| Train with 50% of the training data instead of 100% | ESAM · ScanNet200 | AP: 42.2 -> 40.2 | Hurt |
| Train with 10% of the training data instead of 100% | ESAM · ScanNet200 | AP: 42.2 -> 32.8 | Hurt |
| Replace geometric-aware pooling with average pooling (ablation) | ESAM · ScanNet-25k | AP: 47.2 -> 45.9 | Hurt |
| Use superpoint features only for decoder cross-attention and mask prediction (ablation) | ESAM · ScanNet-25k | AP: 47.2 -> 34.5 | Hurt |
| Use point-wise features only for decoder cross-attention and mask prediction (ablation) | ESAM · ScanNet-25k | AP: 47.2 -> 47.4 | No clear effect |
| Add box representation to merging (ablation) | ESAM · ScanNet | AP: 33.4 -> 41.6 | Helped |
| Add contrastive representation to merging (ablation) | ESAM · ScanNet | AP: 36.9 -> 41.6 | Helped |
Surprising findings
- Offline VFM-assisted methods perform worse on 3RScan, which the authors attribute to blurry RGB frames and poor camera poses from fast-moving cameras.
- With only 10% of the training data, ESAM still matches or exceeds SAI3D's full-data ScanNet200 result (32.8 AP vs 28.2).
Limitations
- Real-time speed depends on the 2D VFM; only FastSAM is real-time, and SAM is not.
- The 3D sparse U-Net and memory adapters are heavy and account for most of the 3D inference time.
- Offline methods underperform on 3RScan due to blurry RGB and poses, indicating sensitivity to sensor quality for the baselines.
Experimental setup
Datasets: ScanNet ScanNet200 SceneNN 3RScan
Benchmarks: ScanNet200 ScanNet SceneNN 3RScan
Metrics: AP AP50 AP25 FPS latency (ms)
Baselines: SAM3D SAMPro3D Open3DIS SAI3D TD3D Oneformer3D INS-Conv TD3D-MA
Builds on: SAM SAM3D Oneformer3D MemAda
Closest papers by meaning
- Online Segment Any 3D Thing as Instance Tracking (NeurIPS 2025 · similarity 0.91)
- OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging (CVPR 2025 · similarity 0.89)
- ODIN: A Single Model for 2D and 3D Segmentation (CVPR 2024 · similarity 0.87)
- Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking (CVPR 2025 · similarity 0.87)
- SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.87)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.