Research map / 3D vision / 3D generation
Denoising Diffusion via Image-Based Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GIBR, a multi-view denoising diffusion model over IB-planes (image-based rendering feature planes) that generates and reconstructs unbounded real-world 3D scenes from 2D images alone, using representation dropout to avoid trivial 3D solutions.
Problem
Generating and reconstructing large, unbounded, real-world 3D scenes from only posed 2D images, where existing generative 3D models are limited to canonically aligned object-centric data, bounded representations, or masks.
Approach
IB-planes: image-space feature maps output jointly by a multi-view U-Net with cross-view attention, lifted into an unbounded neural field via camera-frustum and polar projections and max-pooled across views; training drops out a view's own features when rendering to that view to prevent trivial flat-plane solutions.
Key results
- Single-view reconstruction on CO3D hydrant: GIBR PSNR_D 16.07 vs RenderDiffusion++ 15.70 and PixelNeRF++ 15.06.
- Unconditional generation FID_D on CO3D hydrant 91.9 vs RenderDiffusion++ 185.4 and VSD* 217.0 (lower is better).
- Removing representation dropout raises FID_H from 118.1 to 266.5 on CO3D hydrant, showing trivial-solution failure without it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GIBR instead of the strongest reported baseline | 3D reconstruction · CO3D hydrant, single-view | PSNR_D: 15.70 -> 16.07 | Helped |
| use GIBR instead of the strongest reported baseline | 3D reconstruction · CO3D apple, single-view | PSNR_D: 16.71 -> 18.09 | Helped |
| use GIBR instead of the strongest reported baseline | 3D reconstruction · ShapeNet car, single-view | PSNR_D: 28.00 -> 29.74 | Helped |
| use GIBR instead of RenderDiff++ | 3D generation · CO3D hydrant | FID_D: 185.4 -> 91.9 | Helped |
| use GIBR instead of Viewset Diffusion | 3D generation · ShapeNet car | FID_D: 56.0 -> 62.8 | Hurt |
| add representation dropout (ablation) | GIBR · CO3D hydrant, generation | FID_H: 266.5 -> 118.1 | Helped |
| add representation dropout (ablation) | GIBR · CO3D hydrant, generation | FID_D: 91.9 -> 58.9 | Hurt |
| use IB-planes instead of triplanes (ablation) | GIBR · CO3D hydrant, single-view reconstruction | PSNR_D: 14.55 -> 16.07 | Helped |
| add cross-view attention (ablation) | GIBR · CO3D hydrant, single-view reconstruction | PSNR_D: 14.91 -> 16.07 | Helped |
| use explicit 3D scene representation and rendering (ablation) | GIBR · CO3D hydrant, single-view reconstruction | PSNR_D: 13.51 -> 16.07 | Helped |
| use explicit 3D scene representation and rendering (ablation) | GIBR · CO3D hydrant, generation | FID_D: 36.3 -> 91.9 | Hurt |
| add polar features (ablation) | GIBR · CO3D hydrant, generation | FID_H: 126.5 -> 118.1 | Helped |
Surprising findings
- Viewset Diffusion is the top-performing baseline on ShapeNet cars (FID_D 56.0 vs GIBR 62.8), since canonically aligned objects favor canonical-frame methods.
- Removing polar features slightly improves single-view reconstruction PSNR (16.27 vs 16.07) despite worsening held-out generation FID.
Limitations
- Assumes each scene is static.
- Slower to train than 2D diffusion models due to volumetric rendering.
- Reconstruction metrics are novel-view-synthesis style rather than direct 3D metrics.
Experimental setup
Datasets: MVImgNet CO3D ShapeNet cars
Benchmarks: CO3D MVImgNet ShapeNet
Metrics: PSNR SSIM LPIPS DRC FID
Baselines: RenderDiffusion++ PixelNeRF++ Viewset Diffusion SparseFusion
Builds on: RenderDiffusion PixelNeRF IBRNet K-planes DDIM
Closest papers by meaning
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.91)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.91)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.90)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.90)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.90)
- DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.