Research map / Detection and segmentation / Open-set and few-shot detection
SANeRF-HQ: Segment Anything for NeRF in High Quality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SANeRF-HQ, which lifts prompt-based SAM (or HQ-SAM) masks into a NeRF object field using density-weighted rendering and a ray-pair RGB similarity loss to get sharper, multi-view-consistent 3D segmentation.
Problem
Prompt-based 3D object segmentation in NeRF is inaccurate at boundaries and inconsistent across views when using 2D foundation models like SAM.
Approach
Instead of distilling SAM features into a neural field (which aliases boundaries), cache SAM features per view, decode masks with user prompts, and aggregate the 2D masks into a 3D object identity field attached to a pre-trained NeRF's density, with an added ray-pair RGB similarity loss that encourages boundary-aware consistency.
Key results
- SANeRF-HQ reaches 91.0 mIoU on Mip-NeRF 360 and 95.2 on LLFF, versus 88.8 and 90.6 for SA3D in Table 1.
- On LERF, SANeRF-HQ achieves 90.7 mIoU versus 69.3 for SA3D and 82.9 for raw SAM.
- Using HQ-SAM with a cache container gives the best results in most datasets, e.g., 94.4 mIoU on Mip-NeRF 360 and 96.8 on LLFF in Table 2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SANeRF-HQ instead of the strongest listed baseline | NeRF object segmentation · Mip-NeRF 360 | mIoU: 88.8 -> 91.0 | Helped |
| Use SANeRF-HQ instead of the strongest listed baseline | NeRF object segmentation · LERF | mIoU: 82.9 -> 90.7 | Helped |
| Use SANeRF-HQ instead of the strongest listed baseline | NeRF object segmentation · LLFF | mIoU: 93.7 -> 95.2 | Helped |
| Use SANeRF-HQ instead of the strongest listed baseline | NeRF object segmentation · 3D-FRONT | mIoU: 78.7 -> 89.9 | Helped |
| Use SANeRF-HQ instead of the strongest listed baseline | NeRF object segmentation · Others | mIoU: 88.8 -> 91.1 | Helped |
| Add Ray-Pair RGB loss (ablation) | SANeRF-HQ · Mip-NeRF 360 | mIoU: 91.0 -> 91.3 | Helped |
| Add Ray-Pair RGB loss (ablation) | SANeRF-HQ · LERF | mIoU: 88.3 -> 90.7 | Helped |
| Add Ray-Pair RGB loss (ablation) | SANeRF-HQ · LLFF | mIoU: 95.2 -> 95.8 | Helped |
| Add Ray-Pair RGB loss (ablation) | SANeRF-HQ · 3D-FRONT | mIoU: 89.3 -> 89.9 | Helped |
| Add Ray-Pair RGB loss (ablation) | SANeRF-HQ · Others | mIoU: 90.9 -> 91.1 | Helped |
| Use feature distillation instead of a feature cache (ablation) | SAM feature container · Mip-NeRF 360 | mIoU: 93.6 -> 91.0 | Hurt |
| Use feature distillation instead of a feature cache (ablation) | SAM feature container · LERF | mIoU: 90.3 -> 90.7 | Helped |
Surprising findings
- Feature distillation and feature caching show no significant performance gap across datasets, so compute cost is the main differentiator.
- SA3D with a single-view mask underperforms on LERF (69.3 mIoU) and 3D-FRONT, attributed to error accumulation from self-prompting.
Limitations
- Cache container is constrained by available memory and requires running the encoder for uncached novel views.
- Sparse prompts can leave views without masks when 3D prompt points are occluded.
- Evaluation uses only 24 hand-picked scenes with some manually annotated ground truth.
Experimental setup
Datasets: Mip-NeRF 360 LERF LLFF 3D-FRONT Panoptic Lifting scenes Contrastive Lift Messy Rooms
Benchmarks: Mip-NeRF 360 LERF LLFF 3D-FRONT Others (Panoptic Lifting and Contrastive Lift scenes)
Metrics: mIoU Accuracy
Baselines: SA3D ISRF SAM SAN
Builds on: Segment Anything NeRF SA3D SAN ISRF
Closest papers by meaning
- NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows (CVPR 2024 · similarity 0.87)
- GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding (CVPR 2024 · similarity 0.87)
- NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation (CVPR 2024 · similarity 0.87)
- GO-N3RDet: Geometry Optimized NeRF-enhanced 3D Object Detector (CVPR 2025 · similarity 0.87)
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods (NeurIPS 2025 · similarity 0.86)
- NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.