Research map / Point clouds / Segmentation and detection
ODIN: A Single Model for 2D and 3D Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ODIN, a single transformer/U-Net that segments single RGB images (2D) and posed multiview RGB-D sequences (3D) by alternating within-view 2D fusion and cross-view 3D k-NN fusion, initialized from Mask2Former COCO weights.
Problem
3D instance and semantic segmentation benchmarks (ScanNet, ScanNet200) evaluate on mesh-sampled point clouds, which hides the benefits of 2D pre-training and posed RGB-D input; models consuming sensor RGB-D frames underperform, and 2D and 3D perception are assumed to need distinct architectures.
Approach
A unified architecture that interleaves 2D within-view attention with 3D cross-view k-NN attention, unprojecting 2D tokens to 3D with depth and camera poses and projecting back, distinguishing 2D and 3D tokens only through their positional encodings, so that most parameters (including 2D pre-trained backbones) are shared across 2D and 3D inputs.
Key results
- ODIN-Swin-B reaches 50.0 mAP on ScanNet instance segmentation with sensor RGB-D input, versus 43.9 for Mask3D trained on the same input, and 31.5 vs 15.5 mAP on ScanNet200 with sensor input.
- On ScanNet semantic segmentation, ODIN-Swin-B reaches 77.8 mIoU with sensor RGB-D input, the best among sensor-input methods (DeepViewAgg 71.0).
- Embodied: HELPER + ODIN reaches 47.7 SR / 61.6 GC on TEACh valid-unseen vs. 37.4 / 55.0 for HELPER.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sensor RGB-D point clouds instead of mesh-sampled point clouds | Mask3D on ScanNet instance segmentation · ScanNet | mAP: 55.2% -> 43.9% | Hurt |
| Use ODIN-Swin-B rather than Mask3D with sensor RGB-D point clouds | ScanNet instance segmentation · ScanNet | mAP: 43.9 -> 50.0 | Helped |
| Use ODIN-Swin-B rather than the strongest listed sensor-input baseline | ScanNet semantic segmentation · ScanNet | mIoU: 71.0 -> 77.8 | Helped |
| Use ODIN-Swin-B rather than MAFT | ScanNet200 instance segmentation · ScanNet200 | mAP: 29.2 -> 31.5 | Helped |
| Train ODIN jointly on ScanNet 3D and COCO 2D data (ablation) | ODIN · ScanNet instance segmentation | mAP: 47.8 -> 49.1; 1.3% absolute improvement | Helped |
| Train ODIN jointly on ScanNet 3D and COCO 2D data (ablation) | ODIN · COCO | mAP: 43.6 -> 41.2 | Hurt |
| Add 3D cross-view fusion layers (ablation) | ODIN · ScanNet instance segmentation | mAP: 39.3 -> 47.8; 8.5% mAP drop | Helped |
| Add 3D cross-view fusion layers (ablation) | ODIN · ScanNet semantic segmentation | mIoU: 73.2 -> 73.3 | No clear effect |
| Interleave within-view 2D and cross-view 3D fusion (ablation) | ODIN · ScanNet instance segmentation | mAP: 41.7 -> 47.8; 6.1% mAP drop | Helped |
| Initialize all layers except 3D fusion layers with pretrained weights (ablation) | ODIN · ScanNet instance segmentation | mAP: 42.3 -> 47.8; 5.5% mAP drop | Helped |
| Use pretrained initialization rather than training the entire model from scratch (ablation) | ODIN · ScanNet instance segmentation | mAP: 41.5 -> 47.8; 6.3% mAP drop | Helped |
| Use Swin-B rather than ResNet50 as the backbone (ablation) | ODIN · ScanNet instance segmentation | mAP: 47.8 -> 50.0 | Helped |
Surprising findings
- Removing 3D cross-view fusion barely affects semantic segmentation (73.2 vs 73.3 mIoU) but drops instance mAP by 8.5 points.
- Joint 2D-3D training improved 3D ScanNet mAP by 1.3 points while reducing 2D COCO performance, indicating competition between modalities.
Limitations
- Depends on accurate depth and camera poses; performance drops sharply with noisy inputs.
- Underperforms mesh-sampled-input methods on ScanNet mAP and on S3DIS-style mesh benchmarks; joint 2D-3D training trades off 2D accuracy.
- Scope limited to unified architecture without large-scale joint training on diverse 2D and 3D datasets.
Experimental setup
Datasets: ScanNet ScanNet200 COCO Matterport3D S3DIS AI2THOR ProcTHOR TEACh ALFRED
Benchmarks: ScanNet ScanNet200 Matterport3D S3DIS AI2THOR COCO TEACh ALFRED
Metrics: mAP mAP50 mAP25 mIoU success rate goal condition success rate
Baselines: Mask3D QueryFormer MAFT PBNet SoftGroup MVPNet BPNet DeepViewAgg VMVF OpenMask3D HELPER FILM
Builds on: Mask2Former Mask3D Point Transformer BUTD-DETR
Closest papers by meaning
- Point-SAM: Promptable 3D Segmentation Model for Point Clouds (ICLR 2025 · similarity 0.88)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.88)
- Segment Any 3D Object with Language (ICLR 2025 · similarity 0.88)
- EmbodiedSAM: Online Segment Any 3D Thing in Real Time (ICLR 2025 · similarity 0.87)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.87)
- OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.