Research map / 3D vision / NeRF and radiance fields
NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeRF On-the-go, a plug-and-play module that predicts per-ray uncertainty from DINOv2 features with an SSIM-based loss and decoupled training, removing distractors from casually captured in-the-wild NeRF training images.
Problem
NeRFs trained on casually captured images containing moving distractors (objects, shadows, lighting changes) produce artifacts, especially under high occlusion ratios, and existing distractor-handling methods degrade in complex scenes and converge slowly.
Approach
Predict per-pixel uncertainty with a small MLP on frozen DINOv2 features (with cosine-similarity neighborhood regularization), train the uncertainty head with an SSIM-based loss on dilated patches, and decouple uncertainty training from NeRF reconstruction via gradient detachment.
Key results
- On the On-the-go dataset's high-occlusion Patio-High scene, the method reaches 21.41 PSNR / 0.718 SSIM / 0.235 LPIPS, versus 20.54 PSNR for RobustNeRF; Ours achieves 23.33 PSNR on High Occlusion Spot scene vs 20.65 for RobustNeRF.
- On the RobustNeRF dataset, the method achieves 23.50 PSNR on Android and 21.58 PSNR on Statue, outperforming RobustNeRF and Mip-NeRF 360 + SAM.
- Dilated patch sampling with rate 4 and patch size 32 gives the best ablation result; the method reaches at 25K iterations detail that RobustNeRF only reaches at 250K iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NeRF On-the-go instead of the strongest listed baseline | NeRF novel view synthesis · Corner | PSNR: 23.04 -> 24.22 | Helped |
| use NeRF On-the-go instead of the strongest listed baseline | NeRF novel view synthesis · Spot | PSNR: 21.08 -> 23.33 | Helped |
| use NeRF On-the-go instead of the strongest listed baseline | NeRF novel view synthesis · Mountain | PSNR: 20.20 -> 20.15 | Hurt |
| use NeRF On-the-go instead of the strongest listed baseline | NeRF novel view synthesis · Patio | PSNR: 20.83 -> 20.78 | Hurt |
| use NeRF On-the-go instead of the strongest listed baseline | NeRF novel view synthesis · Fountain | PSNR: 17.20 -> 20.11 | Helped |
| use NeRF On-the-go instead of the strongest listed baseline | NeRF novel view synthesis · Patio-High | PSNR: 20.54 -> 21.41 | Helped |
| use NeRF On-the-go instead of the strongest listed baseline | NeRF novel view synthesis · Statue | PSNR: 21.30 -> 21.58 | Helped |
| use NeRF On-the-go instead of the strongest listed baseline | NeRF novel view synthesis · Android | PSNR: 22.83 -> 23.50 | Helped |
| use dilated patch sampling with dilation rate 4 instead of rates 1, 2, 8, or 16 (ablation) | NeRF On-the-go · Patio-High | PSNR: 17.82, 20.70, 21.41, 18.22, 16.08 | Mixed |
| add uncertainty regularization Lreg (ablation) | NeRF On-the-go · Patio-High | PSNR: 21.02 -> 21.41 | Helped |
| use SSIM instead of L2 loss for uncertainty learning (ablation) | NeRF On-the-go · Patio-High | PSNR: 17.13 -> 21.41 | Helped |
| decouple uncertainty prediction and NeRF training (ablation) | NeRF On-the-go · Patio-High | PSNR: 16.70 -> 21.41 | Helped |
Surprising findings
- Mip-NeRF 360 + SAM and RobustNeRF underperform in low-distractor settings, and RobustNeRF underperforms even when no distractors are present.
- Training the uncertainty MLP jointly with the NeRF causes a large performance drop (PSNR 16.70 vs 21.41), motivating decoupling.
Limitations
- Uncertainty prediction is unreliable for regions with strong view-dependent effects such as reflective windows and metals.
- Relies on a frozen DINOv2 feature extractor as a prior for distractor identification.
- Evaluated on limited set of scenes (6 quantitative sequences from On-the-go dataset).
Experimental setup
Datasets: RobustNeRF dataset On-the-go dataset Mip-NeRF 360 static scenes
Benchmarks: On-the-go Dataset RobustNeRF Dataset Mip-NeRF 360 static scene
Metrics: PSNR SSIM LPIPS
Baselines: Mip-NeRF 360 D2NeRF NeRF-W Ha-NeRF RobustNeRF Mip-NeRF 360 + SAM
Builds on: NeRF in the Wild RobustNeRF Mip-NeRF 360 DINOv2 S3IM
Closest papers by meaning
- NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation (CVPR 2024 · similarity 0.92)
- NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning (AAAI 2024 · similarity 0.91)
- Optimize the Unseen - Fast NeRF Cleanup with Free Space Prior (NeurIPS 2025 · similarity 0.91)
- Rad-NeRF: Ray-decoupled Training of Neural Radiance Field (NeurIPS 2024 · similarity 0.91)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.91)
- LuSh-NeRF: Lighting up and Sharpening NeRFs for Low-light Scenes (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.