Research map / 3D vision / 3D generation
SparseGNV: Generating Novel Views of Indoor Scenes with Sparse RGB-D Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparseGNV, a feed-forward framework that combines a neural point cloud (Point-NeRF) with a transformer that autoregressively generates VQ tokens to synthesize novel views of indoor scenes from sparse RGB-D inputs.
Problem
Novel view synthesis of indoor scenes from sparse RGB-D views, requiring both photorealism for largely unobserved regions and multi-view consistency.
Approach
Marries an explicit neural point cloud that renders geometric guidance images with a VQ-token autoregressive transformer generator conditioned on a scene context (reference and probed previews) and a target query, enabling generalizable feed-forward novel view generation without per-scene optimization.
Key results
- On ScanNet with 2 observed views, SparseGNV reaches 15.713 PSNR / 0.564 SSIM / 0.533 LPIPS, versus 14.281 PSNR for DDP-NeRF (best baseline).
- With 8 observed views, SparseGNV achieves 17.931 PSNR / 0.577 SSIM / 0.463 LPIPS, outperforming DDP-NeRF's 17.491 PSNR.
- Removing the rendered color images (w/o R-Color) drops 2-view PSNR from 15.713 to 13.157, showing the neural point cloud's structural cues matter most.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SparseGNV instead of DDP-NeRF | ScanNet novel-view synthesis · ScanNet test scenes, |O| = 2 | PSNR: 14.281 -> 15.713 | Helped |
| Use SparseGNV instead of ViewFormer | ScanNet novel-view synthesis · ScanNet test scenes, |O| = 2 | SSIM: 0.541 -> 0.564 | Helped |
| Use SparseGNV instead of DDP-NeRF | ScanNet novel-view synthesis · ScanNet test scenes, |O| = 2 | LPIPS: 0.712 -> 0.533 | Helped |
| Use SparseGNV instead of DDP-NeRF | ScanNet novel-view synthesis · ScanNet test scenes, |O| = 4 | PSNR: 15.799 -> 16.622 | Helped |
| Use SparseGNV instead of ViewFormer | ScanNet novel-view synthesis · ScanNet test scenes, |O| = 4 | SSIM: 0.549 -> 0.568 | Helped |
| Use SparseGNV instead of DDP-NeRF | ScanNet novel-view synthesis · ScanNet test scenes, |O| = 4 | LPIPS: 0.630 -> 0.500 | Helped |
| Use SparseGNV instead of DDP-NeRF | ScanNet novel-view synthesis · ScanNet test scenes, |O| = 8 | PSNR: 17.491 -> 17.931 | Helped |
| Use SparseGNV instead of DDP-NeRF | ScanNet novel-view synthesis · ScanNet test scenes, |O| = 8 | SSIM: 0.567 -> 0.577 | Helped |
| Use SparseGNV instead of DDP-NeRF | ScanNet novel-view synthesis · ScanNet test scenes, |O| = 8 | LPIPS: 0.554 -> 0.463 | Helped |
| Add rendered color images to the scene context and query (ablation) | SparseGNV · ScanNet test scenes, |O| = 2, 4, 8 | PSNR: 13.157 -> 15.713; 15.124 -> 16.622; 16.010 -> 17.931 | Helped |
| Add ray maps to previews (ablation) | SparseGNV · ScanNet test scenes, |O| = 2, 4, 8 | PSNR: 15.239 -> 15.713; 15.739 -> 16.622; 16.827 -> 17.931 | Helped |
| Add reference previews to the scene context (ablation) | SparseGNV · ScanNet test scenes, |O| = 2, 4, 8 | PSNR: 14.957 -> 15.713; 15.755 -> 16.622; 17.007 -> 17.931 | Mixed |
Surprising findings
- Removing the probed previews sometimes improves metrics (e.g., 2-view PSNR 15.825 vs 15.713 full model), which the authors attribute to probed previews occasionally misleading generation.
Limitations
- Outputs from VQ-based generation are less 3D consistent than volume-rendering methods; object details and lighting can change.
- Requires accurate camera poses, which may be unavailable when input views are extremely sparse.
- Evaluated only on ScanNet with a small set of five held-out test scenes.
Experimental setup
Datasets: ScanNet
Benchmarks: ScanNet test scenes
Metrics: PSNR SSIM LPIPS
Baselines: Point-NeRF PixelSynth IBRNet ViewFormer DDP-NeRF
Builds on: Point-NeRF VQ-GAN ViewFormer Scene Representation Transformer
Closest papers by meaning
- Sparis: Neural Implicit Surface Reconstruction of Indoor Scenes from Sparse Views (AAAI 2025 · similarity 0.89)
- Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion (NeurIPS 2025 · similarity 0.87)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.87)
- Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis (NeurIPS 2024 · similarity 0.87)
- CM-EVS: Sparse Panoramic RGB-D-Pose Data for Complete Scene Coverage (NeurIPS 2026 · similarity 0.87)
- LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.