Research map / 3D vision / 3D generation
GenFusion: Closing the Loop between Reconstruction and Generation via Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenFusion, a reconstruction-driven video diffusion model (fine-tuned DynamiCrafter with RGB-D VAE on masked-reconstruction artifact/GT pairs) combined with a cyclic reconstruction-generation fusion loop over 2D Gaussian Splatting for artifact-free sparse-view and far-view 3D scene synthesis.
Problem
3D reconstruction needs dense views while 3D generation works from one or no views; reconstructions produce artifacts for far-field viewpoints, and feed-forward and generative methods saturate or lack scene-scale coverage.
Approach
Train a video diffusion model to map artifact-prone RGB-D renderings from masked 3D reconstructions to ground-truth RGB-D video, then iteratively use its restored novel-view videos as extra photometric supervision during 2DGS optimization (a cyclic fusion loop).
Key results
- On Mip-NeRF 360 with 3/6/9 views, GenFusion averages 16.93 PSNR vs 16.87 for ReconFusion and 14.92 for 2DGS.
- On DL3DV at 1/4 fps split, GenFusion reaches 20.01 PSNR / 0.292 LPIPS vs 17.83 / 0.370 for FSGS.
- Fine-tuning with RGB-D VAE at 960x512 resolution improves FID to 22.55 from 26.16 for the RGB VAE baseline at 512x320.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GenFusion instead of ReconFusion | Mip-NeRF360 sparse-view reconstruction · Mip-NeRF360 sparse-view, average (3/6/9 views) | PSNR: 16.87 -> 16.93 | Helped |
| Use GenFusion instead of the strongest baseline, 3DGS | Mip-NeRF360 sparse-view reconstruction · Mip-NeRF360 sparse-view, average (3/6/9 views) | LPIPS: 0.509 -> 0.517 | Hurt |
| Use GenFusion instead of FSGS | Sparse-view reconstruction · DL3DV 1/2 fps | PSNR: 18.25 -> 20.47 | Helped |
| Use GenFusion instead of 3DGS | Sparse-view reconstruction · DL3DV 1/2 fps | LPIPS: 0.314 -> 0.284 | Helped |
| Use GenFusion instead of 2DGS | Sparse-view reconstruction · TnT 1/2 fps | LPIPS: 0.409 -> 0.427 | Hurt |
| Use GenFusion instead of FSGS | Sparse-view reconstruction · DL3DV 1/4 fps | PSNR: 17.83 -> 20.01 | Helped |
| Use GenFusion instead of 3DGS | Sparse-view reconstruction · TnT 1/4 fps | LPIPS: 0.440 -> 0.447 | Hurt |
| Replace RGB-D VAE with RGB VAE (ablation) | GenFusion video diffusion model · Video generation model | FID: 22.5526 -> 26.1607 | Hurt |
| Use 48 frames instead of 16 (ablation) | GenFusion video diffusion model · Video generation model | FID: 22.5526 -> 29.3545 | Hurt |
| Increase resolution from 512×320 to 960×512 (ablation) | GenFusion video diffusion model · Video generation model | FID: 25.4006 -> 22.5526 | Helped |
Surprising findings
- The RGB-D VAE fine-tuned model achieves better FID than the RGB VAE even though the backbone was pretrained on the RGB VAE latent space.
- Gaussian Splatting can be brought to performance comparable to NeRF state of the art in sparse-view Mip-NeRF 360 settings with this approach.
Limitations
- Requires additional denoising steps and about 40 extra minutes of training per scene.
- Filling large invisible regions can cause blurriness from inconsistency between video fragments.
Experimental setup
Datasets: DL3DV-10K Mip-NeRF 360 Tanks and Temples
Benchmarks: Mip-NeRF 360 DL3DV-Benchmark Tanks and Temples
Metrics: PSNR SSIM LPIPS FID
Baselines: Zip-NeRF DiffusioNeRF FreeNeRF SimpleNeRF ZeroNVS ReconFusion 3DGS 2DGS FSGS
Builds on: DynamiCrafter 2D Gaussian Splatting ReconFusion LDM3D DepthSplat
Closest papers by meaning
- GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (CVPR 2025 · similarity 0.88)
- Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion (CVPR 2025 · similarity 0.88)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.88)
- GenRec: Knowing Where to Reconstruct and Where to Generate (NeurIPS 2026 · similarity 0.88)
- Taming Video Diffusion Prior with Scene-Grounding Guidance for 3D Gaussian Splatting from Sparse Inputs (CVPR 2025 · similarity 0.88)
- Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.