Research map / 3D vision / 3D generation
EscherNet: A Generative Model for Scalable View Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EscherNet, a Stable Diffusion-based multi-view conditioned diffusion model with a relative camera positional encoding (CaPE) that performs zero-shot novel view synthesis and single/multi-image 3D reconstruction from any number of reference and target views.
Problem
View synthesis and 3D reconstruction are usually handled by scene-specific neural rendering methods or single-reference 3D diffusion models that fix the number and poses of target views; a generalisable, flexible-view generative 3D representation is lacking.
Approach
Repurposes Stable Diffusion's self-attention (target-to-target) and cross-attention (reference-to-target) as multi-view attention and applies a rotary-style camera positional encoding (CaPE) to queries and keys so attention depends only on relative camera transformations, enabling arbitrary numbers of reference and target views.
Key results
- On GSO-30 with one reference view, EscherNet reaches PSNR 20.24 vs 18.93 for Zero-1-to-3-XL, and 25.90 PSNR with 10 reference views.
- On GSO single-view 3D reconstruction, EscherNet attains Chamfer distance 0.0314 vs 0.0400 for SyncDreamer (about 25% improvement).
- With 10 reference views, EscherNet's Chamfer distance is 0.0167, roughly 60% better than SyncDreamer, and it generates over 100 consistent target views on a single consumer GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EscherNet instead of Zero123-XL with one reference view | Novel view synthesis · GSO-30 | PSNR: 18.93 -> 20.24 | Helped |
| Use EscherNet instead of Zero123-XL with one reference view | Novel view synthesis · GSO-30 | LPIPS: 0.124 -> 0.095 | Helped |
| Use EscherNet instead of Zero123-XL with one reference view | Novel view synthesis · RTMV | PSNR: 10.59 -> 10.56 | No clear effect |
| Increase the number of reference views from 1 to 10 | EscherNet · GSO-30 | PSNR: 20.24 -> 25.90 | Helped |
| Increase the number of reference views from 1 to 10 | EscherNet · RTMV | PSNR: 10.56 -> 15.55 | Helped |
| Use EscherNet instead of InstantNGP with one reference view | Novel view synthesis · NeRF Synthetic | PSNR: 10.92 -> 13.36 | Helped |
| Use EscherNet instead of SyncDreamer with one reference view | 3D reconstruction · GSO | Chamfer distance: 0.0400 -> 0.0314 | Helped |
| Use EscherNet instead of SyncDreamer with one reference view | 3D reconstruction · GSO | Volume IoU: 0.5220 -> 0.5974 | Helped |
Surprising findings
- EscherNet trained with only 3 reference and 3 target views generalises to generating over 100 target views and to any number of reference views.
- Zero-shot EscherNet beats InstantNGP and Gaussian Splatting with fewer than five reference views on NeRF Synthetic, which fail to produce meaningful content in that regime.
Limitations
- Autoregressive generation with many reference views reduces inference time but degrades generation quality.
- Operates in a 3 DoF (object-centric, 4 DoF camera) setting constrained by the training dataset, not general SE(3) poses.
- Scene-specific methods (InstantNGP, 3DGS) surpass it once many reference views are available for NeRF Synthetic.
Experimental setup
Datasets: Objaverse-1.0 Google Scanned Objects RTMV NeRF Synthetic
Benchmarks: Google Scanned Objects (GSO) RTMV NeRF Synthetic GSO 3D reconstruction
Metrics: PSNR SSIM LPIPS Chamfer distance Volume IoU
Baselines: Zero-1-to-3 Zero-1-to-3-XL RealFusion InstantNGP 3D Gaussian Splatting Point-E Shape-E One-2-3-45 DreamGaussian SyncDreamer
Builds on: Stable Diffusion v1.5 Zero-1-to-3 RoPE NeuS
Closest papers by meaning
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.89)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.88)
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.88)
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.88)
- MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model (CVPR 2025 · similarity 0.87)
- NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.