Research map / 3D vision / 3D generation
GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenNBV, an end-to-end RL-trained next-best-view policy that moves in a 5D free space and uses a multi-source state embedding (probabilistic 3D occupancy grid, semantic RGB frames, action history) to generalize from Houses3K to unseen objects.
Problem
Active 3D reconstruction with next-best-view (NBV) policies typically relies on hand-crafted criteria, restricted action spaces (e.g., hemispheres), or per-scene NeRF optimization, which limits generalization to unseen building-scale geometries.
Approach
Extends the NBV action space to 5D free space (3D position plus yaw and pitch) and replaces 2D or per-scene representations with a multi-source state embedding built from a probabilistic 3D occupancy grid, grayscale RGB history, and historical actions, trained end-to-end with PPO on coverage-gain rewards.
Key results
- GenNBV reaches 98.26% final coverage ratio and 91.19% mean AUC on the unseen Houses3K test set, versus 92.20%/86.48% for Scan-RL with its representation and 91.61%/84.49% for Scan-RL.
- On OmniObject3D house category (cross-dataset), GenNBV achieves 97.12% final coverage and 88.63% mean AUC, outperforming Scan-RL's 92.53% and 82.17%.
- On Replica Room 0, GenNBV improves mean AUC from 69.21% (Scan-RL) to 88.53%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GenNBV instead of Scan-RL | Next-Best-View policy · Houses3K Test Set | Final Coverage Ratio: 91.61% -> 98.26% | Helped |
| use GenNBV instead of Scan-RL | Next-Best-View policy · OmniObject3D house category | Final Coverage Ratio: 92.53% -> 97.12% | Helped |
| use multi-source embedding (probabilistic 3D grid, 5-DoF pose, semantic 2D map) instead of probabilistic 3D grid alone (ablation) | GenNBV · Houses3K Test Set | Mean AUC: 81.06% -> 91.19% | Helped |
| use multi-source embedding (probabilistic 3D grid, 5-DoF pose, semantic 2D map) instead of 5-DoF pose alone (ablation) | GenNBV · Houses3K Test Set | Mean AUC: 69.53% -> 91.19% | Helped |
| use multi-source embedding (probabilistic 3D grid, 5-DoF pose, semantic 2D map) instead of semantic 2D map alone (ablation) | GenNBV · Houses3K Test Set | Mean AUC: 81.24% -> 91.19% | Helped |
| add semantic 2D map to probabilistic 3D grid and 5-DoF pose (ablation) | GenNBV · Houses3K Test Set | Mean AUC: 88.66% -> 91.19% | Helped |
| add probabilistic 3D grid to 5-DoF pose and semantic 2D map (ablation) | GenNBV · Houses3K Test Set | Mean AUC: 89.77% -> 91.19% | Helped |
| add 5-DoF pose to probabilistic 3D grid and semantic 2D map (ablation) | GenNBV · Houses3K Test Set | Mean AUC: 88.30% -> 91.19% | Helped |
| use probabilistic 3D grid instead of geometric 2D map (ablation) | depth-based GenNBV representation · Houses3K Test Set | Mean AUC: 71.86% -> 81.06% | Helped |
| use probabilistic 3D grid instead of binary 3D grid (ablation) | depth-based GenNBV representation · Houses3K Test Set | Mean AUC: 77.28% -> 81.06% | Helped |
Surprising findings
- Policies trained only on 2D features (Scan-RL style) did not generalize well across objects, while the 3D grid representation improved results.
- A probabilistic 3D grid outperformed a binary 3D grid, showing that distinguishing unscanned from free cells matters for NBV.
Limitations
- Training restricted to house-like objects from Houses3K with complex internal structures excluded.
- Image resolution downsampled to 400x400 due to memory limits.
- Evaluated only in simulation (Isaac Gym) with a simulated CrazyFlie drone.
Experimental setup
Datasets: Houses3K OmniObject3D Objaverse Replica
Benchmarks: Houses3K test set OmniObject3D house category OmniObject3D animals/trucks/dinosaurs Replica Room 0 Objaverse (qualitative)
Metrics: mean AUC of coverage ratio final coverage ratio Chamfer distance accuracy (cm)
Baselines: Random Random Hemisphere Uniform Hemisphere Uncertainty-Guided (NeRF-based NBV) ActiveRMap Scan-RL
Builds on: PPO Scan-RL CURL Legged Gym
Closest papers by meaning
- GenFusion: Closing the Loop between Reconstruction and Generation via Videos (CVPR 2025 · similarity 0.86)
- Free3D: Consistent Novel View Synthesis without 3D Representation (CVPR 2024 · similarity 0.85)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.85)
- 3D Reconstruction with Generalizable Neural Fields using Scene Priors (ICLR 2024 · similarity 0.85)
- NextBestPath: Efficient 3D Mapping of Unseen Environments (ICLR 2025 · similarity 0.85)
- EscherNet: A Generative Model for Scalable View Synthesis (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.