Research map / 3D vision / 3D generation
GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenWarp, a semantic-preserving generative warping framework that fine-tunes Stable Diffusion 1.5 to perform single-image novel view synthesis by augmenting self-attention with cross-view attention conditioned on warped coordinate embeddings, instead of inpainting explicitly warped images.
Problem
Single-image novel view synthesis for general scenes is limited by noisy monocular depth maps that corrupt explicit warping, and by loss of semantic details from the input view when warping to distant viewpoints in warping-and-inpainting pipelines.
Approach
Instead of inpainting an explicitly warped image, a two-stream SD1.5-based model takes the source view and warped coordinate embeddings from MDE depth; cross-view attention to the source features is concatenated into the self-attention so the network learns where to warp and where to generate.
Key results
- On in-domain RealEstate10K mid-range views, GenWarp reaches FID 31.10 and PSNR 14.55, versus 32.70 / 12.26 for GeoGPT w/depth and 41.76 / 14.21 for SD-Inpainting.
- On out-of-domain ScanNet mid-range views, GenWarp reaches FID 46.03 and PSNR 12.95, versus 52.20 / 11.68 for SD-Inpainting.
- Ablation: warped coordinate embedding gives FID 32.40, better than warped depth map (34.17), warped image (35.27), and Plücker camera embedding (39.10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GenWarp instead of SD-Inpainting | Novel view synthesis · ScanNet mid range | FID: 52.20 -> 46.03 | Helped |
| use GenWarp instead of SD-Inpainting | Novel view synthesis · ScanNet mid range | PSNR: 11.68 -> 12.95 | Helped |
| use GenWarp instead of GeoGPT w/depth | Novel view synthesis · RealEstate10K mid range | FID: 32.70 -> 31.10 | Helped |
| use GenWarp instead of SD-Inpainting fine-tuned | Novel view synthesis · RealEstate10K mid range | PSNR: 14.35 -> 14.55 | Helped |
| use GenWarp instead of GeoGPT w/depth | Novel view synthesis · RealEstate10K long range | FID: 33.91 -> 32.40 | Helped |
| use GenWarp instead of SD-Inpainting fine-tuned | Novel view synthesis · RealEstate10K long range | PSNR: 13.10 -> 13.55 | Helped |
| replace warped coordinate embedding with warped depth map (ablation) | GenWarp · Ablation study | FID: 32.40 -> 34.17 | Hurt |
| replace warped coordinate embedding with warped image (ablation) | GenWarp · Ablation study | FID: 32.40 -> 35.27 | Hurt |
| replace warped coordinate embedding with camera embedding (ablation) | GenWarp · Ablation study | FID: 32.40 -> 39.10 | Hurt |
Surprising findings
- Explicit feature warping with noisy DUSt3R pseudo depth pairs led to unstable training, motivating the implicit warping design.
Limitations
- Struggles with extremely distant camera viewpoints where depth-based correspondence has no influence.
- Quality depends on fine-tuning dataset quality; pseudo depth from DUSt3R is not highly accurate.
- Possible societal biases inherited from multi-view training datasets.
Experimental setup
Datasets: RealEstate10K ScanNet ACID
Benchmarks: RealEstate10K test set ScanNet
Metrics: FID PSNR LPIPS
Baselines: GeoGPT Photometric-NVS SD-Inpainting warping-and-inpainting SD-Inpainting (fine-tuned)
Builds on: Stable Diffusion latent diffusion models Zero123 LucidDreamer Animate-Anyone
Closest papers by meaning
- Learning Visual Generative Priors without Text (CVPR 2025 · similarity 0.85)
- GenRec: Knowing Where to Reconstruct and Where to Generate (NeurIPS 2026 · similarity 0.85)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.85)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.85)
- Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curation (ICML 2025 · similarity 0.85)
- More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.