Research map / 3D vision / NeRF and radiance fields
Rad-NeRF: Ray-decoupled Training of Neural Radiance Field
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rad-NeRF, a ray-decoupled multi-NeRF framework that softly gates an ensemble of Instant-NGP-based sub-NeRFs per ray and regularizes them with depth-based mutual learning to reduce training interference from occluded rays.
Problem
NeRF rendering quality degrades in complex scenes with occlusions and free trajectories, where 3D points invisible to some rays interfere with training; existing multi-NeRF methods rely on manual allocation rules or point-based partitions.
Approach
Decouple NeRF training along the ray dimension with a jointly learned soft gate that assigns per-ray gating scores to sub-NeRFs (shared hash feature grid, independent MLP decoders), fuse outputs after volume rendering, and add depth-based mutual learning where sub-NeRFs regress the gate-fused depth.
Key results
- Rad-NeRF reaches 21.708 PSNR on TAT vs 20.722 for Instant-NGP and 22.061 for MipNeRF360 (trained ~1 day), and 27.871 on NeRF-360-v2 vs 27.309 for Instant-NGP.
- With only 0.2% extra parameters, Rad-NeRF improves rendering performance by up to 1.5 dB over Instant-NGP.
- On ScanNet, Rad-NeRF averages 28.805 PSNR vs 27.984 for Instant-NGP and 28.180 for Block-NGP; 4 sub-NeRFs reach 29.268.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ray-decoupled training with gate-guided fusion and depth mutual learning | Instant-NGP · TAT | PSNR: 20.722 -> 21.708 | Helped |
| apply ray-decoupled training with gate-guided fusion and depth mutual learning | Instant-NGP · NeRF-360-v2 | PSNR: 27.309 -> 27.871 | Helped |
| apply ray-decoupled training with gate-guided fusion and depth mutual learning | Instant-NGP · Free-Dataset | PSNR: 25.951 -> 26.449 | Helped |
| apply ray-decoupled training with gate-guided fusion and depth mutual learning | ZipNeRF · NeRF-360-v2 | PSNR: 28.201 -> 28.492 | Helped |
| add gate-guided fusion instead of uniform fusion (ablation) | Rad-NeRF · TAT (M60, Playground, Train, Truck; Avg) | PSNR: 20.798 -> 21.708 | Helped |
| add depth mutual learning (ablation) | Rad-NeRF · TAT (M60, Playground, Train, Truck; Avg) | PSNR: 21.087 -> 21.708 | Helped |
| use ray-level instead of point-level fusion (ablation) | Rad-NeRF · TAT | PSNR: 20.796 -> 21.708 | Helped |
| use pixel-level instead of image-level fusion (ablation) | Rad-NeRF · TAT | PSNR: 21.503 -> 21.708 | Helped |
| use gate-guided fused depth instead of equal-weight averaged depths as the mutual-learning target (ablation) | Rad-NeRF · TAT (M60, Playground, Train, Truck; Avg) | PSNR: 21.419 -> 21.708 | Helped |
| share the feature grid instead of using independent feature grids (ablation) | Rad-NeRF · TAT (M60, Playground, Train, Truck; Avg) | PSNR: 21.264 -> 21.708 | Helped |
| add depth mutual learning when using uniform fusion (ablation) | Uniform-fusion Rad-NeRF · TAT (M60, Playground, Train, Truck; Avg) | PSNR: 20.798 -> 20.704 | Hurt |
| add CV-balanced gating regularization (ablation) | Rad-NeRF · TAT (M60, Playground, Train, Truck; Avg) | PSNR: 20.545 -> 21.708 | Helped |
Surprising findings
- Sharing the feature grid across sub-NeRFs outperforms independent feature grids; the gain comes mainly from independent MLP decoders.
- Depth-based mutual learning has no benefit under uniform fusion because the averaged depth is inaccurate.
Limitations
- Generalization of the gate module depends on dense training views; performs poorly in few-shot settings (6/9 views show no clear gain).
- Multi-decoder structure reduces inference speed relative to Instant-NGP.
- Not applicable to explicit 3D Gaussian splatting because it requires ray-based neural predictions.
Experimental setup
Datasets: MaskTanks-and-Temples Tanks-and-Temples NeRF-360-v2 Free-Dataset ScanNet LLFF
Benchmarks: MaskTanks-and-Temples Tanks-and-Temples NeRF-360-v2 Free-Dataset ScanNet
Metrics: PSNR SSIM LPIPS
Baselines: NeRF NeRF++ MipNeRF360 DVGO Instant-NGP F2-NeRF Switch-NGP Block-NGP Mega-NGP 3D GS ZipNeRF
Builds on: NeRF Instant-NGP Switch-NeRF Block-NeRF Mega-NeRF
Closest papers by meaning
- Aug-NeRF: Training Stronger Neural Radiance Fields With Triple-Level Physically-Grounded Augmentations (CVPR 2022 · similarity 0.93)
- PyNeRF: Pyramidal Neural Radiance Fields (NeurIPS 2023 · similarity 0.93)
- Ced-NeRF: A Compact and Efficient Method for Dynamic Neural Radiance Fields (AAAI 2024 · similarity 0.92)
- NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning (AAAI 2024 · similarity 0.91)
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.91)
- A General Implicit Framework for Fast NeRF Composition and Rendering (AAAI 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.