Research map / 3D vision / Gaussian splatting
iSegMan: Interactive Segment-and-Manipulate 3D Gaussians
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iSegMan, an interactive 3D Gaussian segment-and-manipulate framework that propagates 2D clicks across views via epipolar constraints and extracts 3D regions with SAM-based visibility-weighted voting, avoiding scene-specific training.
Problem
Controlling the manipulation region in 3D Gaussian Splatting scenes is hard, and existing interactive 3D segmentation tools require scene-specific training that slows down scene manipulation.
Approach
Epipolar-guided Interaction Propagation maps a single 2D click to other views along epipolar lines using DINO feature affinity, and Visibility-based Gaussian Voting treats region extraction as a voting game between 2D pixels and 3D Gaussians weighted by alpha-blended visibility, so no per-scene training is needed; a manipulation toolbox then edits the selected region.
Key results
- On SPIn-NeRF, iSegMan reaches 92.4% mIoU and 99.1% mAcc with no scene-specific training, versus 91.9% mIoU for SA3D, which needs about 5 min feature training.
- On NVOS, iSegMan reaches 92.0% mIoU and 98.4% mAcc, comparable to SAGA (90.9%) and SA3D (90.3%) with lower execution time.
- In semantic editing, iSegMan gets a user study score of 4.52 and CLIP directional similarity 0.2189, above GaussianEditor (3.32, 0.2071) and Instruct-GS2GS (2.10, 0.1647).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iSegMan instead of Instruct-GS2GS for semantic editing | 3D scene semantic editing · Semantic editing | CLIPdir: 0.1647 -> 0.2189 | Helped |
| Use iSegMan instead of SA3D for interactive 3D segmentation | Interactive 3D segmentation · SPIn-NeRF | mIoU (%): 91.9 -> 92.4 | Helped |
| Remove the epipolar constraint (ablation) | iSegMan · Interactive 3D segmentation | mIoU | Hurt |
| Remove the iterative inspection mechanism (ablation) | iSegMan · Interactive 3D segmentation | mIoU | Hurt |
| Change the feature extractor (ablation) | iSegMan · Interactive 3D segmentation | mIoU | No clear effect |
Limitations
- Semantic editing is bounded by the quality of the underlying image editor (InstructPix2Pix).
- Per-interaction latency is limited by gradient-based optimization for semantic editing, so it is not real-time.
- The method relies on SAM 2D masks and DINO features, so errors in them propagate into the 3D selection.
Experimental setup
Datasets: Mip-NeRF 360 Instruct-NeRF2NeRF NVOS SPIn-NeRF LERF LLFF
Benchmarks: SPIn-NeRF NVOS Instruct-NeRF2NeRF
Metrics: mIoU mAcc user study score CLIP directional similarity execution time
Baselines: Instruct-GS2GS GaussianEditor SA3D SAGA ISRF LangSplat MVSeg NVOS
Builds on: 3D Gaussian Splatting Segment Anything InstructPix2Pix DINO Instruct-NeRF2NeRF
Closest papers by meaning
- GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.88)
- Mani-GS: Gaussian Splatting Manipulation with Triangular Mesh (CVPR 2025 · similarity 0.86)
- GaussianPro: 3D Gaussian Splatting with Progressive Propagation (ICML 2024 · similarity 0.86)
- EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting (CVPR 2025 · similarity 0.85)
- 3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting (ICLR 2025 · similarity 0.85)
- Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.