Research map / Video / Video generation
PanoDiT: Panoramic Videos Generation with Diffusion Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PanoDiT, a Diffusion Transformer-based text-to-panoramic-video generator with a DTM-LoRA motion adapter and two panoramic-specific losses (PDLoss, ESLoss), trained on a curated 649-pair PHQ360 dataset.
Problem
Text-to-panoramic (360-degree ERP) video generation is hard for existing U-Net-based T2V models, which struggle with polar high-motion regions, high-frequency detail in low-latitude regions, and long temporal sequences.
Approach
Replace the U-Net denoiser with a DiT backbone that has temporal and global attention, add a LoRA on the DiT motion module for camera-motion control (DTM-LoRA), and add a polar dynamic loss and an equatorial high-frequency stability loss tailored to ERP videos.
Key results
- On PHQ360, PanoDiT achieves FVD 584.36 and FID 75.73, versus SVD at FVD 1032.56 and FID 98.67 and AnimateDiff at FVD 2319.45.
- On WEB360, PanoDiT reaches FVD 761.72 versus 1290.17 for SVD and 1895.29 for 360DVD.
- Ablation: adding ESLoss improves FVD from 641.2 to 517.9 and FID from 80.61 to 75.51.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ESLoss (ablation) | PanoDiT · ESLoss ablation | FID: 80.61 -> 75.51 | Helped |
| add ESLoss (ablation) | PanoDiT · ESLoss ablation | IS: 10.82 -> 12.44 | Helped |
| add ESLoss (ablation) | PanoDiT · ESLoss ablation | FVD: 641.2 -> 517.9 | Helped |
| add ESLoss (ablation) | PanoDiT · ESLoss ablation | PSNR: 10.25 -> 11.07 | Helped |
| add ESLoss (ablation) | PanoDiT · ESLoss ablation | SSIM: 0.58 -> 0.62 | Helped |
| add ESLoss (ablation) | PanoDiT · ESLoss ablation | LPIPS: 0.62 -> 0.59 | Helped |
| add PDLoss (ablation) | PanoDiT · Panoramic video high-latitude regions | motion coherence | Helped |
| apply DTM-LoRA with increased weight (ablation) | PanoDiT · Panoramic video generation | motion score | Helped |
| use PanoDiT instead of SVD | Text-to-panorama generation · WEB360 | FVD: 1290.17 -> 761.72 | Helped |
| use PanoDiT instead of SVD | Text-to-panorama generation · PHQ360 | FVD: 1032.56 -> 584.36 | Helped |
Limitations
- Training set is small (649 PHQ360 pairs), and DiT lacks strong image priors so a high-quality panorama dataset is required.
- Evaluation relies largely on FVD/FID-style metrics and a user study of 20 participants; the paper notes these metrics give a limited view of quality.
- Hardware, training compute, and the exact DiT backbone specifications are not reported.
Experimental setup
Datasets: WEB360 ODV360 PHQ360
Benchmarks: WEB360 PHQ360
Metrics: FID IS FVD PSNR SSIM LPIPS user study confusion matrix Likert viewing ratings
Baselines: AnimateDiff 360DVD SVD
Builds on: Diffusion Transformer AnimateDiff Latent Diffusion Models LoRA 360DVD
Closest papers by meaning
- 360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model (CVPR 2024 · similarity 0.91)
- DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes (CVPR 2025 · similarity 0.90)
- PanoWan: Lifting Diffusion Video Generation Models to 360$^\circ$ with Latitude/Longitude-aware Mechanisms (NeurIPS 2025 · similarity 0.90)
- ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models (NeurIPS 2025 · similarity 0.89)
- DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion (NeurIPS 2024 · similarity 0.88)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.