Research map / 3D vision / 3D generation
GenRec: Knowing Where to Reconstruct and Where to Generate
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenRec, a multi-view flow matching model that splits generative novel view synthesis into a generative backbone for unobserved pixels and a gradient-isolated pixel-space refinement branch for observed pixels, gated by a depth-derived observation mask.
Problem
Generative novel view synthesis from sparse views applies one uniform distribution-matching loss to all pixels, although observed pixels have a near-unique correct value while unobserved pixels admit many plausible completions, hurting fidelity in observed regions.
Approach
A per-pixel observation mask (from source-view forward warping of monocular depth) splits the model: a multi-view flow matching backbone jointly denoises RGB and scene-coordinate latents for all pixels, while a separate pixel-space refinement stage (geometry-aware decoder adapter plus sparse 3D cross-attention) corrects only observed pixels, with masked stop-gradients and no gradient into the back
Key results
- On RealEstate10K single-view extrapolation, GenRec reaches PSNR 17.05 / LPIPS 0.3220 / FID 33.09, versus GEN3C at 15.24 / 0.3624 / 35.68.
- On DL3DV-10K single-view, GenRec reports PSNR 19.80 and FID 32.36, versus GEN3C at 18.47 and 44.54.
- Ablation on RE10K: removing mask gating raises FID from 33.09 to 40.43 and removing gradient isolation from the backbone lowers PSNR from 17.05 to 16.99 (row c).
Surprising findings
- Allowing reconstruction gradients to reach the backbone did not improve reconstruction or generation quality relative to the decoupled design.
- Mask-gated supervision of the unified model (no separate branch) hurt FID substantially (40.43 vs 33.09).
Limitations
- Inherits accuracy limits of monocular depth front-end, which bounds mask and warp quality.
- Training limited to at most two input views per scene due to compute.
- Uniform-view and strong view-dependent effects remain failure modes; very low coverage reduces model to backbone.
Experimental setup
Datasets: RealEstate10K DL3DV-10K Mip-NeRF 360
Benchmarks: RealEstate10K DL3DV-10K Mip-NeRF 360
Metrics: PSNR SSIM LPIPS FID ATE RPE
Baselines: CameraCtrl ViewCrafter SEVA GLD Gen3R GEN3C Geometry Forcing LVSM 3DGS Difix3D+ pixelSplat MVSplat
Builds on: Conditional Flow Matching Diff2Flow Stable Diffusion LPIPS Gen3C
Closest papers by meaning
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.89)
- Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion (NeurIPS 2025 · similarity 0.89)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.89)
- GenFusion: Closing the Loop between Reconstruction and Generation via Videos (CVPR 2025 · similarity 0.88)
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.88)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.