Research map / Point clouds / Segmentation and detection
Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NPISeg3D, a probabilistic interactive 3D segmentation framework using hierarchical neural processes (scene- and object-level latents) and a probabilistic prototype modulator, providing uncertainty maps alongside segmentation from user clicks.
Problem
Interactive 3D segmentation from sparse user clicks needs few-shot generalization to unseen scenes/objects and reliable predictive uncertainty to guide users; existing methods are deterministic.
Approach
Formulates click-based segmentation as a neural process: clicks form the context set and unlabeled points the target set. A hierarchical latent structure (scene latent z_s, object latents z_o) and a stochastic FiLM-style modulator of click prototypes yield probabilistic predictions; variance over latent samples gives uncertainty.
Key results
- On KITTI-360 multi-object segmentation, NPISeg3D reaches 52.9% IoU@15 and 48.5% average IoU versus AGILE3D's 44.3% average IoU.
- On S3DIS multi-object (out-of-domain), IoU@5 of 89.0 vs AGILE3D 86.3 and NoC@80 of 2.8 vs 3.4.
- Ablation: removing the probabilistic modulator (deterministic) drops S3DIS mIoU@10 from 90.9 to 89.6 and KITTI-360 from 48.5 to 46.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NPISeg3D instead of AGILE3D | Multi-object interactive 3D segmentation · KITTI-360 | IoU@5: 40.5 -> 44.0 | Helped |
| Use NPISeg3D instead of AGILE3D | Multi-object interactive 3D segmentation · S3DIS-A5 | NoC@80: 3.4 -> 2.8 | Helped |
| Use NPISeg3D instead of AGILE3D | Single-object interactive 3D segmentation · KITTI-360 | IoU@5: 44.4 -> 55.7 | Helped |
| Add scene-specific and object-specific latent variables (ablation) | NPISeg3D · S3DIS | mIoU@10: 88.7 -> 90.9 | Helped |
| Add scene-specific and object-specific latent variables (ablation) | NPISeg3D · KITTI-360 | mIoU@10: 45.4 -> 48.5 | Helped |
| Add the probabilistic prototype modulator (ablation) | NPISeg3D with both hierarchical latent variables · S3DIS | mIoU@10: 90.4 -> 90.9 | Helped |
| Replace deterministic modulation with probabilistic modulation (ablation) | NPISeg3D prototype modulation · KITTI-360 | mIoU@10: 46.1 -> 48.5 | Helped |
| Use probabilistic Add modulation instead of the proposed modulation (ablation) | NPISeg3D prototype modulation · S3DIS | mIoU@10: 90.4 -> 90.9 | Helped |
| Increase MC dropout rate from r=0.1 to r=0.5 (ablation) | MC dropout uncertainty estimation · KITTI-360 | mIoU@10: 40.6 -> 24.9 | Hurt |
| Increase NPISeg3D latent sampling number from n=5 to n=10 (ablation) | NPISeg3D uncertainty estimation · S3DIS | mIoU@10: 90.8 -> 90.9 | No clear effect |
| Add the previous segmentation mask (ablation) | NPISeg3D · KITTI-360 multi-object segmentation | IoU@10: 47.8 -> 48.5 | Helped |
| Use RITM click simulation instead of iterative click simulation (ablation) | NPISeg3D training · KITTI-360 multi-object segmentation | IoU@15: 50.5 -> 52.9 | Helped |
Surprising findings
- Out-of-domain performance still lags in-domain performance, as acknowledged by the authors.
- MC dropout performance degrades as dropout rate increases (KITTI-360 mIoU@10 falls from 40.6 to 24.9 at r=0.5), whereas NPISeg3D is stable across sample counts.
Limitations
- Out-of-domain performance lags in-domain performance.
- Uncertainty maps are not yet used to select next clicks (left for future work).
Experimental setup
Datasets: ScanNetV2 S3DIS Replica KITTI-360 PartNet
Benchmarks: ScanNetV2-Val S3DIS Replica KITTI-360 PartNet
Metrics: IoU@5 IoU@10 IoU@15 NoC@80 NoC@85 NoC@90 mIoU@10
Baselines: InterObject3D InterObject3D++ AGILE3D MC dropout
Builds on: Neural Processes AGILE3D InterObject3D RITM
Closest papers by meaning
- AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation (ICLR 2024 · similarity 0.89)
- OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging (CVPR 2025 · similarity 0.86)
- TETRIS: Towards Exploring the Robustness of Interactive Segmentation (AAAI 2024 · similarity 0.86)
- NTClick: Achieving Precise Interactive Segmentation With Noise-tolerant Clicks (CVPR 2025 · similarity 0.86)
- GrabS: Generative Embodied Agent for 3D Object Segmentation without Scene Supervision (ICLR 2025 · similarity 0.86)
- Online Segment Any 3D Thing as Instance Tracking (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.