Research map / 3D vision / NeRF and radiance fields
NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Heuristics-Guided Segmentation (HuGS), combining SfM-based and color-residual heuristics as rough static cues with SAM to produce static masks that improve NeRF training in scenes with transient distractors.
Problem
NeRF assumes static scenes, so transient distractors (moving objects, shadows) cause artifacts; existing segmentation-based methods need class priors and heuristic-based methods produce imprecise masks.
Approach
Use hand-crafted heuristics only as rough static cues (SfM feature match frequency for high-frequency textures, color residuals from a partially trained Nerfacto for low-frequency textures) to prompt SAM, which refines them into sharp static/transient masks used as loss weights in NeRF training, without requiring prior class knowledge.
Key results
- On Kubric, HuGS with Mip-NeRF 360 reaches 41.21 average PSNR vs 39.64 for RobustNeRF (prior best) and 29.02 for native Mip-NeRF 360.
- On Kubric segmentation, HuGS achieves 0.944 mIoU and 0.934 F1, above DINO (0.789 mIoU) and Grounded-SAM (0.747 mIoU) without prior knowledge.
- On the Distractor dataset, HuGS with Mip-NeRF 360 averages 27.29 PSNR on Ball/Android-style scenes subset average 27.29 for Baby Yoda scene and 25.66 for Nerfacto variant, improving over native baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply HuGS to Mip-NeRF 360 | Mip-NeRF 360 · Kubric (Avg.) | PSNR: 29.02 -> 41.21 | Helped |
| Apply HuGS to Mip-NeRF 360 | Mip-NeRF 360 · Distractor (Avg.) | PSNR: 23.67 -> 27.29 | Helped |
| Apply HuGS to Mip-NeRF 360 | Mip-NeRF 360 · Phototourism (Avg.) | PSNR: 23.66 -> 24.43 | Helped |
| Add SfM heuristics, color-residual heuristics, and SAM static maps (ablation) | Nerfacto · Kubric (Avg.) | PSNR: 29.76 -> 40.40 | Helped |
| Add SfM heuristics, color-residual heuristics, and SAM static maps (ablation) | Nerfacto · Distractor (Avg.) | PSNR: 22.06 -> 25.66 | Helped |
| Add SfM heuristics to color-residual heuristics and SAM (ablation) | Nerfacto · Kubric (Avg.) | PSNR: 40.12 -> 40.40 | Helped |
| Add SfM heuristics to color-residual heuristics and SAM (ablation) | Nerfacto · Distractor (Avg.) | PSNR: 24.58 -> 25.66 | Helped |
| Add SAM to SfM and color-residual heuristics (ablation) | Nerfacto · Kubric (Avg.) | PSNR: 40.11 -> 40.40 | Helped |
| Add SAM to SfM and color-residual heuristics (ablation) | Nerfacto · Distractor (Avg.) | PSNR: 25.40 -> 25.66 | Helped |
| Add color-residual heuristics to SfM heuristics (ablation) | Nerfacto · Kubric (Avg.) | PSNR: 38.14 -> 40.11 | Helped |
| Add color-residual heuristics to SfM heuristics (ablation) | Nerfacto · Distractor (Avg.) | PSNR: 25.67 -> 25.40 | Hurt |
Surprising findings
- Partially trained (about 5k iteration) Nerfacto residual maps, which give relatively low-quality renderings, still suffice as rough heuristic cues when refined by SAM.
Limitations
- Relies on the assumption that rough heuristics about static objects are accurate enough to prompt SAM.
- Requires SfM (COLMAP) and a partially trained NeRF, adding preprocessing cost.
- Gains are smaller on Phototourism where landmark bodies are little affected by distractors.
Experimental setup
Datasets: Kubric Distractor dataset (RobustNeRF) Phototourism dataset (NeRF-W)
Benchmarks: Kubric Distractor dataset Phototourism dataset
Metrics: PSNR SSIM LPIPS mIoU F1
Baselines: Nerfacto Mip-NeRF 360 NeRF-W HA-NeRF D2NeRF RobustNeRF DeepLabv3+ Mask2Former Grounded-SAM DINO
Builds on: NeRF-W RobustNeRF Nerfacto Mip-NeRF 360 Segment Anything
Closest papers by meaning
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.92)
- Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (CVPR 2022 · similarity 0.91)
- Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation From 2D Supervision (CVPR 2023 · similarity 0.90)
- NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning (AAAI 2024 · similarity 0.90)
- BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling (AAAI 2024 · similarity 0.89)
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.