Research map / 3D vision / 3D generation
CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
CubeDiff generates 360° panoramas from text and/or an image by fine-tuning a pretrained Stable Diffusion-style latent diffusion model to jointly denoise six cubemap faces with inflated attention layers.
Problem
Generating high-quality, seamlessly wrapping 360° panoramas is hard due to scarce panorama data, equirectangular distortion, and the need for cross-view consistency, with prior methods relying on complex architectures or autoregressive outpainting.
Approach
Represent panoramas as six 90° perspective cubemap faces and jointly denoise them by inflating every 2D attention layer of a pretrained text-to-image LDM across the six faces, with synchronized GroupNorm in the VAE, UV positional encodings, and 2.5° face overlaps, requiring no correspondence-aware attention.
Key results
- On Laval Indoor, CubeDiff (img+multitxt) reports FID 10.0 and CLIP-FID 4.1 versus MVDiffusion's FID 25.7 and CLIP-FID 13.5.
- In a 2AFC user study with 28 participants, the no-text variant won 19.5% of preferences versus 12.3% for Diffusion360 and 7.0% for MVDiffusion.
- Scaling the training set from 700 (tiny) to ~40k panoramas improves Laval FID from 27.3 to 10.0 (full model).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CubeDiff img+txt instead of MVDiffusion | Panorama generation · LAVAL Indoor | FID: 25.7 -> 9.5 | Helped |
| Use CubeDiff img+multitxt instead of PanFusion | Panorama generation · SUN360 | FID: 30.0 -> 24.1 | Helped |
| Use CubeDiff img+multitxt instead of MVDiffusion | Panorama generation · LAVAL Indoor | CS: 27.44 -> 30.17 | Helped |
| Use CubeDiff img+multitxt instead of MVDiffusion | Panorama generation · SUN360 | CS: 25.54 -> 28.14 | Helped |
| Add synchronized GroupNorm across spatial and frame dimensions (ablation) | CubeDiff · Panorama generation | visual quality | Helped |
| Add 2.5° overlap to cube-face predictions (ablation) | CubeDiff · Panorama generation | boundary continuity | Helped |
| Reduce training data from the full set to approximately 700 panoramas (ablation) | CubeDiff · LAVAL Indoor | FID: 10.0 -> 27.3 | Hurt |
| Reduce training data from the full set to approximately 700 panoramas (ablation) | CubeDiff · SUN360 | FID: 24.1 -> 41.7 | Hurt |
| Reduce training data from over 40,000 panoramas to about 20,000 (ablation) | CubeDiff · LAVAL Indoor | FID: 10.0 -> 13.8 | Hurt |
| Reduce training data from over 40,000 panoramas to about 20,000 (ablation) | CubeDiff · SUN360 | FID: 24.1 -> 23.9 | No clear effect |
Surprising findings
- A tiny model trained on only ~700 Polyhaven panoramas remains competitive, with Laval FID 27.3 and KID 1.05 similar to baselines.
Limitations
- Cubemap faces cause discontinuities at edges requiring overlap-and-crop handling.
- Evaluation uses Laval Indoor and SUN360, which overlap with some baselines' training data, so the comparison is not fully clean.
- Metrics are computed on perspective renderings because perceptual networks are trained on perspective images.
Experimental setup
Datasets: Polyhaven Humus Structured3D Pano360 Laval Indoor SUN360
Benchmarks: Laval Indoor SUN360
Metrics: FID KID CLIP-FID FAED CLIP score user preference (2AFC)
Baselines: Text2Light PanFusion OmniDreamer PanoDiffusion Diffusion360 MVDiffusion
Builds on: Stable Diffusion Latent diffusion models Classifier-free guidance v-prediction MVDiffusion
Closest papers by meaning
- Taming Stable Diffusion for Text to 360 Panorama Image Generation (CVPR 2024 · similarity 0.92)
- DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion (NeurIPS 2024 · similarity 0.89)
- HORIZON: High-Resolution Semantically Controlled Panorama Synthesis (AAAI 2024 · similarity 0.88)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.87)
- Conditional Panoramic Image Generation via Masked Autoregressive Modeling (NeurIPS 2025 · similarity 0.87)
- ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.