Research map / 3D vision / 3D generation
OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Lifts multi-view inconsistent SAM 2D masks into a consistent NeRF-based 3D feature field using hierarchical contrastive learning, enabling category-agnostic hierarchical and multi-object 3D segmentation.
Problem
Category-agnostic 3D scene segmentation that segments any object without limits on count or category while reflecting hierarchical part-whole structure, given multi-view 2D segmentations that are inconsistent across views.
Approach
A hierarchical 2D representation (disjoint patches from one-hot mask membership plus a voting-based correlation matrix) is used to supervise a NeRF feature field with a contrastive loss that pulls same-patch features together and orders related patches by hierarchy level with a per-level decay.
Key results
- On the Replica hierarchical benchmark, OmniSeg3D reaches 90.1% average mIoU versus 86.5% for SAM, with level-2 mIoU of 88.9% vs 80.2%.
- On NVOS, OmniSeg3D achieves 91.7% mIoU vs 90.3% for SA3D; on MVSeg it achieves 94.3% vs 92.4% for SA3D.
- Hierarchical decay lambda=0.5 gives the best average mIoU (90.1%), trading level-1 accuracy (91.3%) for level-2 gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hierarchical modeling to the contrastive learning method (ablation) | OmniSeg3D · Replica hierarchical segmentation Level 2 | mIoU (%): 80.4 -> 88.9 | Helped |
| add hierarchical modeling to the contrastive learning method (ablation) | OmniSeg3D · Replica hierarchical segmentation Level 1 | mIoU (%): 93.1 -> 91.3 | Hurt |
| set per-level decay λ to 0.1 rather than omit hierarchical modeling (ablation) | OmniSeg3D · Replica hierarchical segmentation | mIoU (%): 86.7 -> 88.6 | Helped |
| set per-level decay λ to 0.2 rather than omit hierarchical modeling (ablation) | OmniSeg3D · Replica hierarchical segmentation | mIoU (%): 86.7 -> 89.4 | Helped |
| set per-level decay λ to 0.5 rather than omit hierarchical modeling (ablation) | OmniSeg3D · Replica hierarchical segmentation | mIoU (%): 86.7 -> 90.1 | Helped |
| set per-level decay λ to 1 rather than omit hierarchical modeling (ablation) | OmniSeg3D · Replica hierarchical segmentation | mIoU (%): 86.7 -> 89.2 | Helped |
| increase feature dimension from 4 to 8 (ablation) | OmniSeg3D · Replica room-0 | Average mIoU: 89.8 -> 91.8 | Helped |
| increase feature dimension from 8 to 16 (ablation) | OmniSeg3D · Replica room-0 | Average mIoU: 91.8 -> 93.0 | Helped |
| increase feature dimension from 16 to 32 (ablation) | OmniSeg3D · Replica room-0 | Average mIoU: 93.0 -> 93.0 | No clear effect |
| increase feature dimension from 32 to 64 (ablation) | OmniSeg3D · Replica room-0 | Average mIoU: 93.0 -> 93.1 | No clear effect |
| increase feature dimension from 64 to 128 (ablation) | OmniSeg3D · Replica room-0 | Average mIoU: 93.1 -> 93.2 | No clear effect |
| use OmniSeg3D instead of SA3D | 3D instance segmentation · NVOS | mIoU (%): 90.3 -> 91.7 | Helped |
Surprising findings
- Plain SAM gives slightly higher level-1 mIoU (92.8%) than OmniSeg3D (91.3%), so the hierarchy gain comes at a small cost on fine-grained parts.
Limitations
- Hierarchy levels lack a clear definition, so one clustering threshold may not segment objects at the same level.
- Contrastive learning is applied per image, so objects never co-occurring in one image may get similar features.
Experimental setup
Datasets: Replica LERF 360-Counter Blender-Lego LLFF-Flower Tanks and Temples NVOS MVSeg
Benchmarks: Replica hierarchical segmentation NVOS MVSeg Replica instance segmentation
Metrics: mIoU accuracy
Baselines: SAM DINO LSeg NVOS ISRF SA3D MVSeg
Builds on: InstantNGP Segment Anything ContrastiveLift Semantic-NeRF contrastive clustering SA3D
Closest papers by meaning
- OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation (NeurIPS 2025 · similarity 0.86)
- Hierarchy-Agnostic Unsupervised Segmentation: Parsing Semantic Image Structure (NeurIPS 2024 · similarity 0.86)
- OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities (CVPR 2025 · similarity 0.86)
- UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior (NeurIPS 2024 · similarity 0.85)
- Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding (AAAI 2025 · similarity 0.85)
- ODIN: A Single Model for 2D and 3D Segmentation (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.