Research map / 3D vision / NeRF and radiance fields
3D Reconstruction with Generalizable Neural Fields using Scene Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NFPs, a generalizable neural scene prior trained from single-view RGB-D images in a two-stage (geometry then texture) paradigm, which enables fusion-free merging of frames and fast per-scene fine-tuning for scene reconstruction and novel view synthesis.
Problem
Per-scene neural field reconstruction requires training from scratch for each scene and many views, is slow, and generalizable multi-view methods rely on learned fusion modules that limit flexibility and scalability.
Approach
Learn generalizable neural fields from single-view RGB-D inputs using a two-stage geometric-then-texture prior over a compact keypoint surface representation with distance-aware interpolation; frames are fused by direct volumetric merging without learnable fusion, then fine-tuned per scene.
Key results
- Feed-forward Ours-prior without per-scene optimization reaches F-score 0.737 on ScanNet mesh reconstruction using about 40 frames, versus 0.747 for MonoSDF and 0.602 for ManhattanSDF.
- With per-scene optimization (15 min), Ours reaches F-score 0.954 on ScanNet, best among compared methods, and Chamfer comp 0.017.
- On ScanNet novel view synthesis, Ours achieves PSNR 26.88 and SSIM 0.909, above NeRF (24.04) and Go-surf (25.47), with NeRFusion slightly higher in LPIPS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add geometric prior (ablation) | NFP · ScanNet surface reconstruction | Acc: 0.079 -> 0.046 | Helped |
| add geometric prior (ablation) | NFP · ScanNet surface reconstruction | Comp: 0.031 -> 0.030 | Helped |
| add geometric prior (ablation) | NFP · ScanNet surface reconstruction | F-score: 0.851 -> 0.862 | Helped |
| add color prior (ablation) | NFP · ScanNet novel view synthesis | PSNR: 25.87 -> 26.88 | Helped |
| add color prior (ablation) | NFP · ScanNet novel view synthesis | SSIM: 0.899 -> 0.909 | Helped |
| add color prior (ablation) | NFP · ScanNet novel view synthesis | LPIPS: 0.415 -> 0.246 | Helped |
| fine-tune the NFP prior per scene | NFP · ScanNet surface reconstruction | F-score: 0.737 -> 0.954 | Helped |
| use NFP instead of Go-surf with 30 frames | surface reconstruction · 10 synthetic scenes, 30 frames | F-score: 0.9042 -> 0.9311 | Helped |
| use NFP instead of NeRFusion | novel view synthesis · ScanNet | PSNR: 26.49 -> 26.88 | Helped |
| use NFP instead of NeRFusion | novel view synthesis · ScanNet | SSIM: 0.915 -> 0.909 | Hurt |
| use NFP instead of NeRFusion | novel view synthesis · ScanNet | LPIPS: 0.209 -> 0.244 | Hurt |
Surprising findings
- The feed-forward prior without any per-scene optimization achieves mesh reconstruction comparable to RGB-based per-scene methods while running hundreds of times faster.
Limitations
- Feed-forward prior alone has lower accuracy (Acc 0.084) than per-scene optimized methods on ScanNet.
- Ground-truth ScanNet meshes are noisy, which makes evaluation challenging.
- Requires posed RGB-D input images.
Experimental setup
Datasets: ScanNetV2 10 synthetic scenes (Neural-RGBD synthetic set)
Benchmarks: ScanNetV2 synthetic RGB-D scenes
Metrics: Acc Comp Prec Recall F-score C-l1 NC PSNR SSIM LPIPS
Baselines: ManhattanSDF MonoSDF NeuRIS HelixSurf Neural-RGBD Go-surf NeRF NSVF NerfingMVS IBRNet NeRFusion BundleFusion
Builds on: NeuS PointConv NeRF Go-surf MonoSDF
Closest papers by meaning
- Learning Robust Generalizable Radiance Field with Visibility and Feature Augmented Point Representation (ICLR 2024 · similarity 0.88)
- Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly (NeurIPS 2024 · similarity 0.88)
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.88)
- NC-SDF: Enhancing Indoor Scene Reconstruction Using Neural SDFs with View-Dependent Normal Compensation (CVPR 2024 · similarity 0.88)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.88)
- Learning Neural Exposure Fields for View Synthesis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.