Research map / Video / Video generation
Panacea: Panoramic and Controllable Video Generation for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Panacea generates panoramic multi-view driving videos from BEV layout sequences and text prompts using a two-stage latent diffusion pipeline with decomposed 4D attention, and the synthetic data improves StreamPETR on nuScenes.
Problem
Generating high-quality, temporally and cross-view consistent, controllable multi-view driving videos with annotations to augment training data for video-based BEV perception.
Approach
Extends Stable Diffusion into a decomposed 4D attention UNet (intra-view spatial, cross-view adjacent-view, and cross-frame temporal attention), uses a two-stage image-then-video pipeline, and injects BEV layout sequences via ControlNet alongside CLIP text prompts.
Key results
- Panacea reaches FVD 139 and FID 16.96 on nuScenes validation, beating DriveDreamer (FVD 452, FID 52.6) and BEVControl (FID 24.85).
- StreamPETR trained on real plus generated data reaches NDS 49.2, +2.3 over real-only (46.9).
- Elevating image-only data to video with Panacea raises NDS by 5.8 points (34.3 to 40.1) while mAP drops 2.3 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Panacea-generated training videos to real training data | StreamPETR · nuScenes validation | NDS: 46.9 -> 49.2 (+2.3 points) | Helped |
| elevate image-only data to video using Panacea | StreamPETR · nuScenes validation | NDS: 34.3 -> 40.1 (+5.8%) | Helped |
| elevate image-only data to video using Panacea | StreamPETR · nuScenes validation | mAP: 28.6 -> 26.3 (-2.3%) | Hurt |
| add cross-view attention (ablation) | Panacea · nuScenes validation | FVD: 247 -> 139 (108) | Helped |
| add cross-view attention (ablation) | Panacea · nuScenes validation | FID: 22.07 -> 16.96 (5.11) | Helped |
| add cross-frame attention (ablation) | Panacea · nuScenes validation | FVD: 214 -> 139 (75) | Helped |
| add cross-frame attention (ablation) | Panacea · nuScenes validation | FID: 20.43 -> 16.96 (3.47) | Helped |
| use the two-stage pipeline instead of direct video generation (ablation) | Panacea · nuScenes validation | FVD: 305 -> 139 (166) | Helped |
| use the two-stage pipeline instead of direct video generation (ablation) | Panacea · nuScenes validation | FID: 36.61 -> 16.96 (19.65) | Helped |
Surprising findings
- Elevating image-only data to video improves NDS by 5.8 points but lowers mAP by 2.3 points, which the authors attribute to generated-versus-real domain mismatch.
Limitations
- Generation is limited to 256x512 resolution with 8-frame clips.
- Controllability is measured only indirectly via a perception model; generated data trained alone reaches only 77% of real-data NDS.
- Mismatch between generated and real data lowers mAP when elevating image datasets to video.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes
Metrics: FID FVD View-Matching-Score NDS mAP mAOE mAVE
Baselines: BEVGen BEVControl DriveDreamer StreamPETR-S
Builds on: Stable Diffusion ControlNet Video Latent Diffusion Model StreamPETR
Closest papers by meaning
- Generating Multimodal Driving Scenes via Next-Scene Prediction (CVPR 2025 · similarity 0.88)
- DriveScape: High-Resolution Driving Video Generation by Multi-View Feature Fusion (CVPR 2025 · similarity 0.87)
- DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation (AAAI 2025 · similarity 0.87)
- Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models (CVPR 2024 · similarity 0.87)
- SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Control (AAAI 2025 · similarity 0.86)
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.