Research map / 3D vision / Gaussian splatting
MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Feed-forward 360° novel view synthesis from as few as 5 sparse views by rendering multi-channel Gaussian features from a MVSplat-based 3DGS model into the latent space of Stable Video Diffusion, which refines the coarse renders end-to-end.
Problem
Feed-forward novel view synthesis of large real-world scenes from sparse, widely displaced views (wide-sweeping or 360°), where minimal overlap and missing content make regression-based 3DGS methods fail.
Approach
Render learned Gaussian features directly into SVD's latent space (instead of RGB images encoded by a frozen encoder) so the diffusion loss backpropagates into the feed-forward geometry backbone, with a latent alignment loss and a local-group view interaction and farthest-point view selection.
Key results
- On DL3DV-10K (n=300), MVSplat360 reaches 16.81 PSNR, 0.514 SSIM, 0.175 DISTS and 17.01 FID, versus 15.72 PSNR and 78.95 FID for MVSplat.
- On RealEstate10K extrapolation, MVSplat360 achieves 1.79 FID and 0.104 DISTS, improving over latentSplat (2.79 FID) and MVSplat (3.83 FID).
- Ablation: conditioning SVD on end-to-end trainable Gaussian features improves FID from 22.78 (ctx-attn) to 17.01 on DL3DV-10K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MVSplat360 instead of latentSplat | DL3DV-10K n=300 · DL3DV-10K n=300 | FID: 37.68 -> 17.01 | Helped |
| use MVSplat360 instead of latentSplat | DL3DV-10K n=150 · DL3DV-10K n=150 | FID: 34.55 -> 18.89 | Helped |
| use MVSplat360 instead of MVSplat | RealEstate10K interpolation · RealEstate10K interpolation | PSNR: 26.39 -> 26.41 | Helped |
| use MVSplat360 instead of MVSplat | RealEstate10K extrapolation · RealEstate10K extrapolation | SSIM: 0.812 -> 0.810 | Hurt |
| add SVD refinement conditioned on rendered images (ablation) | MVSplat · DL3DV-10K | SSIM: 0.433 -> 0.399 | Hurt |
| add SVD refinement conditioned on rendered images (ablation) | MVSplat · DL3DV-10K | FID: 78.95 -> 38.05 | Helped |
| average CLIP tokens from multiple context views (ablation) | MVSplat360 + SVD · DL3DV-10K | SSIM: 0.399 -> 0.467 | Helped |
| render Gaussian features into diffusion latent space and train end-to-end (ablation) | MVSplat360 + ctx-attn · DL3DV-10K | FID: 22.78 -> 17.01 | Helped |
| increase test-time input views from 5 to 7 (ablation) | MVSplat360 · DL3DV-10K | SSIM: 0.514 -> 0.526 | Helped |
| reduce test-time input views from 5 to 3 (ablation) | MVSplat360 · DL3DV-10K | SSIM: 0.514 -> 0.432 | Hurt |
| upscale inputs 2× with bilinear interpolation before encoding (ablation) | MVSplat360 · DL3DV-10K test set | PSNR: 26.77dB -> 32.52dB | Helped |
Surprising findings
- Naively conditioning SVD on MVSplat's rendered RGB outputs performs slightly worse than the original MVSplat in PSNR-related metrics (SSIM 0.399 vs 0.433).
- Even with 3 input views at test time, the model outperforms regression baselines (pixelSplat and MVSplat) that use 5 views.
Limitations
- Outputs can be oversaturated, which limits pixel-aligned metric gains, inherited from SVD; color histogram matching is used as a workaround.
- Diffusion refinement can hallucinate content not present in the input views.
- Inference is slow due to multiple diffusion sampling steps.
Experimental setup
Datasets: DL3DV-10K RealEstate10K
Benchmarks: DL3DV-10K RealEstate10K
Metrics: PSNR SSIM LPIPS DISTS FID
Baselines: pixelSplat MVSplat latentSplat PixelNeRF Du et al.
Builds on: MVSplat Stable Video Diffusion pixelSplat latentSplat 3D Gaussian Splatting
Closest papers by meaning
- Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion (NeurIPS 2025 · similarity 0.92)
- LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias (ICLR 2025 · similarity 0.90)
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.90)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.90)
- MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model (CVPR 2025 · similarity 0.89)
- Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.