Research map / 3D vision / NeRF and radiance fields
AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AltNeRF, which alternates self-supervised monocular depth/pose estimation with NeRF training so that depth and pose priors regularize and initialize NeRF, improving novel view synthesis from monocular videos without known poses.
Problem
NeRF trained from sparse or video views without accurate camera poses and with only 2D supervision often converges to incorrect geometry and poor novel views, especially for textureless or view-limited scenes.
Approach
Couples a self-supervised monocular depth and pose estimator (SPM) with a BARF-style NeRF (SRM) in an alternating loop: SPM provides depth regularization and pose initialization to NeRF, and NeRF's refined depth and poses are distilled back into SPM, with an error-tolerant depth loss, residual pose refinement, warmup learning, and a multi-view consistency weighting.
Key results
- On LLFF, AltNeRF reaches 27.41 PSNR / 0.803 SSIM / 0.139 LPIPS, beating NeRF (26.36 / 0.778 / 0.147).
- On ScanNet depth estimation, AltNeRF reduces Abs Rel to 0.051 from 0.075 for the second best method (DS-NeRF and NerfingMVS).
- On Captures, AltNeRF reaches 29.72 PSNR and 0.067 LPIPS, improving over DS-NeRF's 27.35 PSNR and 0.109 LPIPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SPM pose-prior initialization (ablation) | BARF · Flower and Scene_02 | PSNR: 23.99 -> 25.21; 29.68 -> 31.31 | Helped |
| add error-tolerant depth regularization (ablation) | BARF with pose prior · Flower and Scene_02 | PSNR: 25.21 -> 24.99; 31.31 -> 34.09 | Mixed |
| use one SPM–SRM alternation (ablation) | BARF with pose and depth priors · Flower and Scene_02 | PSNR: 24.99 -> 25.96; 34.09 -> 35.05 | Helped |
| increase alternations from one to two (ablation) | AltNeRF · Flower and Scene_02 | PSNR: 25.96 -> 26.07; 35.05 -> 35.05 | Helped |
| increase alternations from two to four (ablation) | AltNeRF · Flower and Scene_02 | PSNR: 26.07 -> 26.13; 35.05 -> 35.05 | Mixed |
| add SPM pose and depth priors, error-tolerant depth regularization, and warmup learning (ablation) | BARF · Scene_04 | camera pose error | Helped |
| use AltNeRF instead of the strongest reported baseline | Novel view synthesis methods · LLFF and Captures | PSNR: 26.36 -> 27.41; 27.35 -> 29.72 | Helped |
| use AltNeRF instead of the strongest reported baseline | Depth estimation methods · ScanNet | Abs Rel: 0.075 -> 0.051 | Helped |
Surprising findings
- Simply combining BARF with DS-NeRF (DS-BARF) using COLMAP poses and depth is insufficient, performing worse than the proposed method.
- BARF with identity initialization fails badly on CO3D (e.g., 9.42 PSNR on Couch_193), showing sensitivity to pose initialization.
Limitations
- Each scene requires long per-scene optimization (4.0-6.4 hours on a single RTX 3090).
- Performance depends on the accuracy of the depth prior; inaccurate depth can impair scene representation until alternation corrects it.
Experimental setup
Datasets: LLFF CO3D ScanNet NYUv2 VOID Captures
Benchmarks: LLFF Captures CO3D ScanNet
Metrics: PSNR SSIM LPIPS Abs Rel Sq Rel RMSE delta accuracy thresholds
Baselines: NeRF BARF NeRFmm DS-NeRF DDP-NeRF NoPe-NeRF DS-BARF NerfingMVS
Builds on: BARF SfM-Learner MonoDepth2 DS-NeRF NerfingMVS
Closest papers by meaning
- Joint Optimization of Neural Radiance Fields and Continuous Camera Motion from a Monocular Video (CVPR 2025 · similarity 0.92)
- FrugalNeRF: Fast Convergence for Extreme Few-shot Novel View Synthesis without Learned Priors (CVPR 2025 · similarity 0.90)
- Few-Shot Neural Radiance Fields under Unconstrained Illumination (AAAI 2024 · similarity 0.89)
- ColNeRF: Collaboration for Generalizable Sparse Input Neural Radiance Field (AAAI 2024 · similarity 0.89)
- ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization (CVPR 2024 · similarity 0.89)
- CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.