Research map / Image generation / Autoregressive and latent generation
PanoDiffusion: 360-degree Panorama Outpainting via Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PanoDiffusion, a latent diffusion model for indoor 360-degree RGB-D panorama outpainting that trains bi-modal on RGB+depth, outpaints RGB without depth at inference, and enforces wraparound consistency via per-step 90-degree rotations.
Problem
Outpainting complete 360-degree indoor panoramas from narrow field-of-view or partially visible RGB inputs, with semantically rich content, diverse outputs, wraparound consistency at panorama ends, and plausible depth; GAN-based and existing diffusion inpainting methods perform poorly on large masks
Approach
A bi-modal latent diffusion model trained on concatenated RGB and depth VQ latents, which at inference needs only RGB input (depth channel replaced with noise); plus camera-rotation augmentation and a two-end alignment mechanism that rotates latents and mask by 90 degrees at every denoising step.
Key results
- On camera-mask RGB outpainting, PanoDiffusion reaches FID 21.55 vs. 31.70 for BIPS (about 67% relative FID improvement over the best baseline).
- With fully masked RGB input, the RGB-D model reaches FID 21.55 vs. BIPS 68.79 on the Structured3D test set, and it stays stable across fully visible, partially visible, and fully masked depth inputs.
- Depth synthesis with partially visible input gives RMSE 219 and Delta1.25 0.9278, the best among compared methods on RMSE and Delta1.25 in that setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the RGB-D bi-modal model rather than no depth during training (ablation) | PanoDiffusion · Structured3D camera mask | FID: 24.33 -> 21.55 | Helped |
| Use the RGB-D bi-modal model rather than no depth during training (ablation) | PanoDiffusion · Structured3D camera mask | density: 0.667 -> 0.867 | Helped |
| Train with 50% sparse depth instead of no depth (ablation) | PanoDiffusion · Structured3D camera mask | FID: 24.33 -> 21.65 | Helped |
| Train with 30% sparse depth instead of no depth (ablation) | PanoDiffusion · Structured3D camera mask | FID: 24.33 -> 21.78 | Helped |
| Train with 10% sparse depth instead of no depth (ablation) | PanoDiffusion · Structured3D camera mask | FID: 24.33 -> 21.68 | Helped |
| Make depth fully visible at inference instead of partially visible (ablation) | PanoDiffusion · Structured3D panorama outpainting | FID: 22.34 -> 21.90 | Hurt |
| Add two-end rotation during denoising (ablation) | PanoDiffusion · Structured3D camera mask | RGB-LRCE: 125.82 -> 90.41 | Helped |
| Add two-end rotation during denoising (ablation) | PanoDiffusion · Structured3D NFoV mask | RGB-LRCE: 128.33 -> 89.74 | Helped |
| Add two-end rotation during denoising (ablation) | PanoDiffusion · Structured3D layout mask | RGB-LRCE: 128.10 -> 88.01 | Helped |
| Add two-end rotation during denoising (ablation) | PanoDiffusion · Structured3D random box mask | RGB-LRCE: 128.19 -> 85.04 | Helped |
Surprising findings
- Giving fully visible depth at inference slightly degraded PanoDiffusion results rather than improving them, which the authors attribute to SNR imbalance between RGB and depth during outpainting.
- Training with sparser depth maps (50%) still improved RGB outpainting over the no-depth LDM baseline.
Limitations
- Rotation-based alignment is applied at latent level, which can introduce extra errors during decoding and leave residual seam differences.
- Evaluated only on indoor Structured3D data at 512x1024 with a two-stage LDM plus super-resolution GAN pipeline.
- Two-stage design (downscaled LDM then GAN upscaler) adds complexity; the RGB-D joint model does not clearly improve over the depth-free model in all settings.
Experimental setup
Datasets: Structured3D
Benchmarks: Structured3D
Metrics: FID sFID density coverage RMSE MAE AbsREL Delta1.25 RGB-LRCE
Baselines: BIPS OmniDreamer LaMa TFill Inpaint Anything RePaint NLSPN CSPN
Builds on: latent diffusion models DDPM RePaint BIPS
Closest papers by meaning
- Spherical-Nested Diffusion Model for Panoramic Image Outpainting (ICML 2025 · similarity 0.87)
- CamFreeDiff: Camera-free Image to Panorama Generation with Diffusion Model (CVPR 2025 · similarity 0.87)
- Taming Stable Diffusion for Text to 360 Panorama Image Generation (CVPR 2024 · similarity 0.86)
- Unboxed: Geometrically and Temporally Consistent Video Outpainting (CVPR 2025 · similarity 0.85)
- CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation (ICLR 2025 · similarity 0.85)
- CM-EVS: Sparse Panoramic RGB-D-Pose Data for Complete Scene Coverage (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.