Research map / 3D vision / 3D generation

DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion

NeurIPS 2024 · 3D vision › 3D generation · approach: Text-to-3D scene generation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a panoramic video-text dataset from HM3D via Habitat and proposes DiffPano, a LoRA-tuned Stable Diffusion for single-view text-to-panorama plus a spherical epipolar-aware attention module for consistent multi-view ERP panorama generation from text and camera poses.

Problem

Generating multi-view consistent 360° equirectangular panoramas from text descriptions and camera poses, which lacks suitable datasets and methods because perspective multi-view methods do not transfer to panoramic projections.

Approach

Derives a spherical epipolar line formulation for ERP panoramas and uses it to define a cross-attention module that samples points along rays and reprojects them onto reference panorama features, combined with LoRA fine-tuning of Stable Diffusion on a new panorama dataset.

Key results

• Pano-SD achieves CS 28.98 (best) and inference time 5.1 s vs PanFusion 27.6 s and Text2Light 50.4 s, with FID 48.52 vs PanFusion 47.62.
• In a user study with about 50 volunteers, DiffPano scores 4.3 on multi-view consistency vs MVDream 3.8 and PanFusion 3.0.
• Ablation over reference views and sample points shows K=2 with S=10 gives the lowest FID (66.02) while K=3,S=10 gives the best PSNR among K=3 settings (32.29), with consistency improving as K and S grow up to S=12 where consistency drops.

Measured effects reported in the paper

Surprising findings

• Two-stage training yields higher FID (74.08 vs 82.92 for one-stage is lower) and lower IS (3.13 vs 4.52) than one-stage, trading diversity for reduced ghosting.
• The authors note that increasing sampling points beyond S=10 to S=12 reduces multi-view consistency.

Limitations

• As the number of frames increases at inference, the model tends to hallucinate content.
• Image quality is slightly inferior due to dataset quality; multi-view panoramas are generated at 256x512 resolution only.
• MVDream adaptation uses full rotation+translation poses, which may hamper that baseline.

Experimental setup

Datasets: HM3D Matterport3D

Benchmarks: PanFusion dataset (MP3D skybox) HM3D-rendered panorama dataset

Metrics: FID IS CLIP score PSNR SSIM LPIPS inference time user study scores

Baselines: Text2Light PanFusion MVDream MVDiffusion

Builds on: Stable Diffusion LoRA EpiDiff MVDream Light Field Networks

Closest papers by meaning

• CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation (ICLR 2025 · similarity 0.89)
• Taming Stable Diffusion for Text to 360 Panorama Image Generation (CVPR 2024 · similarity 0.89)
• Conditional Panoramic Image Generation via Masked Autoregressive Modeling (NeurIPS 2025 · similarity 0.89)
• ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models (NeurIPS 2025 · similarity 0.89)
• PanoWan: Lifting Diffusion Video Generation Models to 360$^\circ$ with Latitude/Longitude-aware Mechanisms (NeurIPS 2025 · similarity 0.89)
• PanoDiT: Panoramic Videos Generation with Diffusion Transformer (AAAI 2025 · similarity 0.88)