Research map / 3D vision / Gaussian splatting
Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Unified-Lift, an end-to-end 2D-to-3D instance segmentation lifting method on 3D Gaussian Splatting that uses a learnable object-level codebook plus association and noisy-label filtering modules, avoiding pre- and post-processing.
Problem
Lifting multi-view 2D instance masks into 3D Gaussian scenes: existing end-to-end methods use direct matching and perform poorly, while two-stage methods need complex pre-/post-processing (e.g., HDBSCAN clustering) that is hyperparameter-sensitive and accumulates error.
Approach
Augment each Gaussian with a contrastively learned feature and add a learnable global object-level codebook whose softmax association with rendered features directly yields instance IDs; codebook learning is stabilized by area-aware ID mapping, a concentration loss, and uncertainty-based noisy label filtering.
Key results
- On LERF-Mask, Unified-Lift reaches 80.9 mIoU and 77.1 mBIoU, above the best prior method (Gaga, 74.7 mIoU).
- On Replica, it reaches 41.6 mIoU and 43.9 F-score, versus 39.1 mIoU for the post-processed OmniSeg3D-GS.
- On Messy Rooms, it averages 69.0 PQscene, comparable to the NeRF-based Contrastive Lift (69.0) and far above the end-to-end Panoptic-Lifting-GS (58.6), with about 1 hour training versus 20+ hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Unified-Lift instead of Gaga | 3D scene segmentation · LERF-Mask | mIoU: 74.7 vs 80.9 | Helped |
| use Unified-Lift instead of Gaga | 3D scene segmentation · LERF-Mask | mBIoU: 72.2 vs 77.1 | Helped |
| use Unified-Lift instead of OmniSeg3D-GS | 3D scene segmentation · Replica | mIoU: 39.1 vs 41.6 | Helped |
| use Unified-Lift instead of OmniSeg3D-GS | 3D scene segmentation · Replica | F-score: 35.9 vs 43.9 | Helped |
| add the object-level codebook (ablation) | end-to-end lifting baseline · Replica | mIoU: 25.3 -> 29.5 | Helped |
| add the object-level codebook (ablation) | end-to-end lifting baseline · Replica | F-score: 32.9 -> 39.2 | Helped |
| add the concentration constraint (ablation) | codebook baseline · Replica | mIoU: 29.5 -> 36.3 | Helped |
| add the concentration constraint (ablation) | codebook baseline · Replica | F-score: 39.2 -> 41.3 | Helped |
| add area-aware ID mapping (ablation) | codebook baseline with concentration constraint · Replica | mIoU: 36.3 -> 39.2 | Helped |
| add area-aware ID mapping (ablation) | codebook baseline with concentration constraint · Replica | F-score: 41.3 -> 41.0 | Hurt |
| add noisy label filtering (ablation) | codebook baseline with concentration constraint and area-aware ID mapping · Replica | mIoU: 39.2 -> 41.6 | Helped |
| add noisy label filtering (ablation) | codebook baseline with concentration constraint and area-aware ID mapping · Replica | F-score: 41.0 -> 43.9 | Helped |
Surprising findings
- The naive codebook baseline (29.5 mIoU on Replica) barely beats the previous end-to-end baseline gap, showing that naive codebook learning is insufficient without the proposed modules.
- Removing the normalization term in the ID mapping (area-aware mapping) is what gives the stable multi-view pseudo-labels, which the authors argue is more important than the sparsity term alone.
Limitations
- Relies on SAM 2D masks and chosen granularity; 2D mask quality bounds performance.
- Replica evaluation uses a customized split because the Gaussian Grouping data was not publicly available.
- Per-scene optimization with a fixed maximum of 256 objects in the codebook.
Experimental setup
Datasets: LERF-Mask Replica Messy Rooms
Benchmarks: LERF-Mask Replica Messy Rooms
Metrics: mIoU mBIoU F-score PQscene
Baselines: Panoptic-Lifting-GS OmniSeg3D-GS Gaussian Grouping Gaga LangSplat LERF Panoptic Lifting Contrastive Lift
Builds on: 3D Gaussian Splatting Panoptic Lifting OmniSeg3D-GS Gaussian Grouping SAM
Closest papers by meaning
- Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting (NeurIPS 2025 · similarity 0.88)
- GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.88)
- InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception (CVPR 2025 · similarity 0.87)
- Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D (CVPR 2024 · similarity 0.87)
- Efficient Decoupled Feature 3D Gaussian Splatting via Hierarchical Compression (CVPR 2025 · similarity 0.86)
- Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.