Research map / 3D vision / NeRF and radiance fields
Joint Optimization of Neural Radiance Fields and Continuous Camera Motion from a Monocular Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Jointly optimizes NeRF and camera poses from a monocular video without priors by modeling continuous camera motion (angular velocity and velocity) and a time-dependent SDF-based NeRF.
Problem
Joint estimation of camera poses and NeRF from monocular video, which existing methods solve only with good pose initialization or depth priors and which struggle with large camera rotations.
Approach
Instead of optimizing frame-wise camera-to-world poses, predict continuous angular and linear velocities with an MLP, integrate them into relative motions, and train a time-dependent SDF NeRF on neighboring frames with flow, photometric and SDF consistency losses; then fix poses and fine-tune a standard NeRF.
Key results
- On Co3D, pose errors are reduced by at least 83% (trajectory ATE), 63% (relative translation) and 83% (relative rotation) versus baselines.
- Depth square relative error reduced by at least 82% on Co3D and 78% on ScanNet relative to compared methods.
- Novel-view synthesis is significantly better than NeRFmm and NoPe-NeRF and comparable to CF3DGS (e.g., Co3D Teddy PSNR 33.04 vs 24.13 for CF3DGS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Lflow (ablation) | the proposed method · Co3D ablation | AbRel: 0.084 -> 0.031 | Helped |
| add Lsdf (ablation) | the proposed method · Co3D ablation | AbRel: 0.072 -> 0.031 | Helped |
| add Lphoto (ablation) | the proposed method · Co3D ablation | AbRel: 0.399 -> 0.031 | Helped |
| use a time-dependent NeRF instead of a conventional NeRF (ablation) | the proposed method · Co3D ablation | AbRel: 0.287 -> 0.031 | Helped |
| fine-tune the time-dependent NeRF instead of training a new conventional NeRF from scratch (ablation) | the proposed method · Co3D ablation | AbRel: 0.082 -> 0.031 | Helped |
| use a motion network instead of optimizing camera poses as learnable variables (ablation) | the proposed method · Co3D ablation | AbRel: 0.080 -> 0.031 | Helped |
Surprising findings
- Learning a conventional NeRF from scratch with the learned poses slightly improves rendering quality but increases depth error by roughly 160%.
Limitations
- Relies on visual cues, so joint optimization struggles in scenes with large low-texture regions.
- Struggles with reflective surfaces.
- Depth and novel-view evaluation rely on up-to-scale recovery using ground-truth scale.
Experimental setup
Datasets: Co3D ScanNet
Benchmarks: Co3D ScanNet
Metrics: PSNR SSIM LPIPS AbRel SqRel delta1 RPEt RPEr ATE
Baselines: NeRFmm NoPe-NeRF CF3DGS
Builds on: NeRF NeuS NoPe-NeRF BARF NeRFmm SDF-flow scene flow constraint
Closest papers by meaning
- NoPe-NeRF: Optimising Neural Radiance Field With No Pose Prior (CVPR 2023 · similarity 0.94)
- D-NeRF: Neural Radiance Fields for Dynamic Scenes (CVPR 2021 · similarity 0.92)
- AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose Optimization (AAAI 2024 · similarity 0.92)
- CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning (AAAI 2024 · similarity 0.91)
- ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization (CVPR 2024 · similarity 0.91)
- BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.