Research map / 3D vision / 3D generation
NTO3D: Neural Target Object 3D Reconstruction with Segment Anything
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NTO3D, which lifts multi-view SAM masks into a 3D occupancy field iteratively prompted by its own projections to segment a user-specified object, then distills SAM encoder features into a 3D feature field to improve neural surface reconstruction of that object.
Problem
Neural 3D reconstruction methods reconstruct whole scenes, while reconstructing a user-indicated target object requires multi-view consistent segmentation that is labor-intensive to obtain; SAM works only on single 2D views.
Approach
A 3D occupancy field, trained with a ray-wise max-based loss to match SAM masks across views, is projected back to 2D to generate new SAM prompts in an iterative loop until convergence; SAM encoder features are then distilled into a 3D feature field rendered alongside color and SDF to boost reconstruction quality.
Key results
- On DTU, NTO3D reaches mean Chamfer distance 0.73, versus 0.84 for NeuS and 0.77 for the no-mask NeuS variant in the paper's table.
- Mask IoU of SAM masks refined by the 3D occupancy field averages 0.983 across 15 DTU scans, compared with 0.945 for DINO masks from SPIn-NeRF.
- With masks, NTO3D achieves mean PSNR 33.06 and SSIM 0.952 on DTU novel view synthesis, versus 30.81 PSNR for NeuS trained with masks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NTO3D iterative mask lifting instead of SPIn-NeRF DINO masks | DTU segmentation · DTU, 15-scene mean | IoU: 0.945 -> 0.983 | Helped |
| use NTO3D rather than NeuS with ground-truth masks | DTU novel view synthesis · DTU, 15-scene mean | PSNR: 30.81 -> 33.06 | Helped |
| use NTO3D rather than NeuS with ground-truth masks | DTU novel view synthesis · DTU, 15-scene mean | SSIM: 0.942 -> 0.952 | Helped |
| use NTO3D rather than NeuS with masks | DTU reconstruction · DTU, 15-scene mean | Chamfer distance: 0.77 -> 0.73 | Helped |
| add a 3D occupancy field (ablation) | Instant-NSR · DTU, 15-scene mean | PSNR: 29.94 -> 32.44 | Helped |
| add a 3D occupancy field (ablation) | Instant-NSR · DTU, 15-scene mean | Chamfer distance: 0.82 -> 0.76 | Helped |
| add a 3D SAM feature field to the 3D occupancy field variant (ablation) | NTO3D · DTU, 15-scene mean | PSNR: 32.44 -> 33.06 | Helped |
| add a 3D SAM feature field to the 3D occupancy field variant (ablation) | NTO3D · DTU, 15-scene mean | Chamfer distance: 0.76 -> 0.73 | Helped |
| distill SAM features at finetuning rather than pretraining (ablation) | NTO3D · DTU, random scene | Chamfer distance: 0.360 -> 0.353 | Helped |
| distill SAM features during iterative occupancy-field training rather than at finetuning (ablation) | NTO3D · DTU, random scene | Chamfer distance: 0.353 -> 0.358 | Hurt |
| vary the initial view used to prompt the target (ablation) | NTO3D · DTU, random scene, initial-view choices | Chamfer distance: 0.731; 0.742; 0.735; 0.740; 0.738 | No clear effect |
Surprising findings
- Distilling SAM features at pretraining or iterative stages performs slightly worse than distilling during finetuning on the target object, since background information leaks into features without the foreground mask.
Limitations
- If SAM fails to segment the target object from the given prompts, the 3D occupancy field cannot be learned.
- Evaluated mainly on DTU and LLFF scenes; BlendedMVS results only in supplementary material.
- Relies on a per-scene SDF neural field and iterative SAM calls, adding training cost.
Experimental setup
Datasets: DTU LLFF BlendedMVS
Benchmarks: DTU LLFF BlendedMVS
Metrics: IoU PSNR SSIM Chamfer distance
Baselines: NeuS NeRF UNISURF IDR COLMAP SPIn-NeRF ISRF SA3D
Builds on: NeuS Instant-NSR Segment Anything (SAM) NeRF
Closest papers by meaning
- Total-Decom: Decomposed 3D Scene Reconstruction with Minimal Interaction (CVPR 2024 · similarity 0.86)
- 3D Reconstruction with Generalizable Neural Fields using Scene Priors (ICLR 2024 · similarity 0.85)
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.85)
- SANeRF-HQ: Segment Anything for NeRF in High Quality (CVPR 2024 · similarity 0.85)
- SAM2Object: Consolidating View Consistency via SAM2 for Zero-Shot 3D Instance Segmentation (CVPR 2025 · similarity 0.85)
- EmbodiedSAM: Online Segment Any 3D Thing in Real Time (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.