Research map / 3D vision / 3D generation
UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UMAMI, a hybrid NVS model that pairs a bidirectional transformer with a deterministic regression head for visible regions and a masked-autoregressive diffusion head for unseen regions, using confidence-gated hybrid sampling.
Problem
Sparse-view novel view synthesis: deterministic networks render observed regions fast but blur unobserved areas, while diffusion-based generative NVS hallucinates plausible content but is slow and expensive to train and sample.
Approach
Factorize the target distribution into a deterministic part (tokens rendered in one forward pass by a regression head, gated by a pixel-wise confidence map) and a stochastic part (remaining tokens sampled by a lightweight MAR-style diffusion head), trained end-to-end on photometric, confidence and diffusion losses with a transformer initialized from LVSM.
Key results
- On Re10K-2view-extra, UMAMI reaches 28.95 PSNR and 0.897 SSIM, the best among compared methods, vs. 28.51 PSNR for LVSM.
- On Re10K-3view, UMAMI reports 31.06 PSNR and 0.946 SSIM, the best among compared methods.
- Rendering takes about 5 s per image at tau=0.95, versus about 1 minute for SEVA, though purely deterministic tau=0 runs in 0.02 s but with blurrier LPIPS (0.398).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UMAMI instead of LVSM | Re10K-2View-Extra · Re10K-2View-Extra | PSNR: 28.51 -> 28.95 | Helped |
| Use UMAMI instead of LVSM | Re10K-2View-Interp · Re10K-2View-Interp | PSNR: 29.67 -> 28.85 | Hurt |
| Use UMAMI instead of LVSM | Re10K-3View · Re10K-3View | PSNR: 30.04 -> 31.06 | Helped |
| Use UMAMI instead of SEVA | DL3DV-1View · DL3DV-1View | PSNR: 13.01 -> 12.81 | Hurt |
| Use UMAMI instead of SEVA | DL3DV-3View · DL3DV-3View | PSNR: 15.95 -> 16.37 | Helped |
| Use UMAMI instead of SEVA | DL3DV-6View · DL3DV-6View | PSNR: 17.98 -> 17.33 | Hurt |
| Increase threshold τ from 0 to 1 (ablation) | UMAMI sampling · sampling ablation | LPIPS: 0.398 -> 0.377 | Helped |
| Increase threshold τ from 0 to 1 (ablation) | UMAMI sampling · sampling ablation | Time (s): 0.02 -> 7.63 | Hurt |
| Increase context views from 1 to 6 (ablation) | UMAMI · DL3DV view-count ablation | LPIPS: 0.574 -> 0.326 | Helped |
| Use random initialization instead of LVSM pretrained weights (ablation) | UMAMI · DL3DV-1View | PSNR: 12.81 -> 11.80 | Hurt |
| Use random initialization instead of LVSM pretrained weights (ablation) | UMAMI · DL3DV-1View | LPIPS: 0.574 -> 0.543 | Helped |
| Generate 3 target views rather than 1 (ablation) | UMAMI · Re10K-2View-Extra | PSNR: 28.95 -> 28.65 | Hurt |
Surprising findings
- On Re10K-2view-interp, LVSM (29.67 PSNR) outperforms UMAMI (28.85 PSNR), so the hybrid is not best on every split.
- On DL3DV-1view, SEVA (13.01 PSNR) beats UMAMI on the metric table, and UMAMI's SSIM (0.269) lags behind several baselines.
Limitations
- Operates in pixel space with many tokens (32x32=1024 per image), increasing memory and compute.
- Does not use pretrained text-to-image priors, limiting hallucination quality.
- Artifacts appear when target camera poses are far from input views; code is proprietary and not released.
Experimental setup
Datasets: RealEstate10K DL3DV
Benchmarks: RealEstate10K DL3DV
Metrics: PSNR SSIM LPIPS
Baselines: MVSplat DepthSplat LVSM ViewCrafter SEVA NoPoSplat
Builds on: LVSM Masked Autoregressive (MAR) DDPM Plücker ray embeddings
Closest papers by meaning
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.92)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.90)
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.90)
- Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion (NeurIPS 2025 · similarity 0.89)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.89)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.