Research map / Image generation / Text-to-image diffusion
Taming Stable Diffusion for Text to 360 Panorama Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PanFusion, a dual-branch Stable Diffusion-based model (panorama branch plus perspective branch linked by equirectangular-perspective projection attention) that generates 360° text-to-panorama images and supports room-layout conditioning.
Problem
Text-to-360° panorama generation is hard due to scarce paired text-panorama data and the geometric/domain gap between equirectangular panoramas and perspective images, with prior multi-view methods producing repetitive elements and global inconsistency.
Approach
Jointly denoise a panorama latent and multiple icosahedron perspective latents with shared SD UNet weights (LoRA-adapted), linked by an EPPA cross-attention module using spherical positional encoding, projection-based attention masks, and bijective weight sharing, plus circular padding for loop consistency.
Key results
- PanFusion reaches FAED 6.04 and panorama FID 46.47 versus 7.19 / 51.69 for SD+LoRA and 97.24 / 76.5 for Text2Light.
- On 20 random perspective views, PanFusion FID is 17.04 versus 19.32 for SD+LoRA and 36.25 for Text2Light.
- In layout-conditioned generation, PanFusion improves 3D IoU from 61.06 (MVDiffusion + ControlNet) to 68.46.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PanFusion instead of SD+LoRA | Text-to-panorama generation · Matterport3D panoramas | FAED: 7.19 -> 6.04 | Helped |
| Use PanFusion instead of SD+LoRA | Text-to-panorama generation · Matterport3D panoramas | IS: 4.40 -> 4.36 | Hurt |
| Add joint initialization of panorama and perspective latent maps (ablation) | PanFusion · Matterport3D panoramas | FID: 59.43 -> 46.47 | Helped |
| Add spherical positional encoding to EPPA (ablation) | PanFusion · Matterport3D panoramas | FAED: 6.63 -> 6.04 | Helped |
| Add the EPP attention mask (ablation) | PanFusion · Matterport3D panoramas | FAED: 6.77 -> 6.04 | Helped |
| Share parameters between the two EPPA directions (ablation) | PanFusion · Matterport3D panoramas | FAED: 7.36 -> 6.04 | Helped |
| Add a perspective branch and EPPA to the panorama branch (ablation) | Pano Branch · Matterport3D panoramas | FAED: 7.90 -> 6.04 | Helped |
| Condition generation on room layout with PanFusion rather than MVDiffusion | Layout-conditioned panorama generation · Matterport3D room layouts | 3D IoU: 61.06 -> 68.46 | Helped |
Surprising findings
- Joint latent map initialization alone worsens FID relative to the Pano Branch baseline in ablation (59.43 vs 50.40), though it helps other metrics.
- Removing the attention mask improves perspective FID (15.81) but hurts panorama FAED.
Limitations
- Higher computational complexity due to the dual-branch architecture.
- Sometimes fails to generate entrances for indoor scenes.
- Trained mostly on indoor scenes (Matterport3D).
Experimental setup
Datasets: Matterport3D BLIP-2 generated captions
Benchmarks: Matterport3D
Metrics: FAED FID Inception Score CLIP Score 3D IoU 2D IoU
Baselines: MVDiffusion Text2Light SD+LoRA Pano Branch
Builds on: Stable Diffusion MVDiffusion LoRA ControlNet Text2Light
Closest papers by meaning
- CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation (ICLR 2025 · similarity 0.92)
- DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion (NeurIPS 2024 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.88)
- Conditional Panoramic Image Generation via Masked Autoregressive Modeling (NeurIPS 2025 · similarity 0.88)
- ViewFusion: Towards Multi-View Consistency via Interpolated Denoising (CVPR 2024 · similarity 0.88)
- StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.