Research map / Image generation / Autoregressive and latent generation
Panorama Generation From NFoV Image Done Right
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Distort-CLIP (a distortion-tuned CLIP metric, Distort-FID) and PanoDecouple, a decoupled diffusion model with a DistortNet for distortion guidance and a ContentNet for content completion, for NFoV-to-panorama outpainting.
Problem
Existing NFoV-to-panorama outpainting methods are evaluated with InceptionNet/CLIP metrics that reward visual quality over panoramic distortion accuracy, and single networks learn distortion and content jointly, leading to "visual cheating" where distortion accuracy degrades.
Approach
Fine-tune CLIP contrastively on distortion types to build a distortion-aware metric (Distort-CLIP/Distort-FID), and decouple panorama generation into a distortion-map-conditioned DistortNet (with condition injection in all ControlNet blocks) and a perspective-image-conditioned ContentNet fused into a frozen U-Net, plus a distortion correction loss using Distort-CLIP.
Key results
- On SUN360, PanoDecouple reaches FID 62.19 and Distort-FID 0.92, best image quality and second-best distortion accuracy (OmniDreamer Distort-FID 0.52).
- On zero-shot Laval Indoor, PanoDecouple achieves FID 74.45 and Distort-FID 2.85, state-of-the-art image quality with strong distortion accuracy.
- Replacing single-block condition registration with all-block DistortNet registration lowers Distort-FID from 2.68 (PanoDiff baseline) to 1.05 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decouple distortion guidance and content completion using PanoDecouple | panorama generation from NFoV images · SUN360 | Distort-FID: 0.52 -> 0.92; 2.68 -> 0.92 | Mixed |
| add distortion-map conditioning to the first block only (ablation) | PanoDecouple · SUN360 | Distort-FID: 2.68 -> 1.63 | Helped |
| register distortion-map conditioning in all blocks rather than only the first block (ablation) | PanoDecouple · SUN360 | Distort-FID: 1.63 -> 1.05 | Helped |
| replace text embedding with perspective image embedding in ContentNet (ablation) | PanoDecouple · SUN360 | IS: 6.35 -> 6.95 | Helped |
| add distortion correction loss (ablation) | PanoDecouple · SUN360 | Distort-FID: 0.98 -> 0.92 | Helped |
| use attention U-Net condition registration instead of the proposed additive DistortNet registration (ablation) | PanoDecouple · SUN360 | Distort-FID: 0.92 -> 2.52 | Hurt |
| use original ControlNet condition registration instead of the proposed additive DistortNet registration (ablation) | PanoDecouple · SUN360 | Distort-FID: 0.92 -> 1.59 | Hurt |
| use attention with original ControlNet condition registration (ablation) | PanoDecouple · SUN360 | Distort-FID: 0.92 -> 5.22 | Hurt |
| use attention with all-block DistortNet registration instead of additive registration (ablation) | PanoDecouple · SUN360 | Distort-FID: 0.92 -> 1.48 | Hurt |
| replace distortion map with a full-zero map (ablation) | PanoDecouple · SUN360 | Distort-FID: 0.92 -> 2.40 | Hurt |
| replace distortion map with a learnable map (ablation) | PanoDecouple · SUN360 | Distort-FID: 0.92 -> 1.42 | Hurt |
| remove first-order Taylor extension from the distortion map (ablation) | PanoDecouple · SUN360 | Distort-FID: 0.92 -> 2.48 | Hurt |
Surprising findings
- Prior methods (e.g., 2S-ODIS, PanoDiff) improve FID while degrading distortion accuracy, a 'visual cheating' phenomenon; OmniDreamer (2022) had the best distortion accuracy.
- Standard InceptionNet and CLIP similarity metrics poorly distinguish panoramic distortion, with InceptionNet only 0.229 similarity between panorama pairs of same distortion.
Limitations
- Trained on only a subset of SUN360 (3000/500 split) since the official full link is unavailable.
- Distort-CLIP evaluates distortion only and does not require generalization, so metric scope is narrow.
Experimental setup
Datasets: SUN360 Laval Indoor
Benchmarks: SUN360 Laval Indoor
Metrics: FID CLIP-FID Distort-FID IS
Baselines: OmniDreamer ImmerseGAN PanoDiff AOG-Net 2S-ODIS
Builds on: ControlNet PanoDiff Latent diffusion CLIP Text2Light
Closest papers by meaning
- ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models (NeurIPS 2025 · similarity 0.87)
- 4K4DGen: Panoramic 4D Generation at 4K Resolution (ICLR 2025 · similarity 0.86)
- 360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model (CVPR 2024 · similarity 0.85)
- HORIZON: High-Resolution Semantically Controlled Panorama Synthesis (AAAI 2024 · similarity 0.85)
- Splatter-360: Generalizable 360 Gaussian Splatting for Wide-baseline Panoramic Images (CVPR 2025 · similarity 0.84)
- DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.