Research map / Video / Video generation
Glad: A Streaming Scene Generator for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Glad generates driving video frame-by-frame by propagating the denoised latent of the previous frame as the noise prior for the next frame of a Stable Diffusion model, with a streaming data sampler for efficient training on nuScenes.
Problem
Existing driving scene generators are either NeRF/3DGS methods that fail on unseen trajectories or diffusion models that produce fixed-length offline videos with high memory use, so they cannot flexibly generate arbitrary-length videos that follow a simulated trajectory.
Approach
Use the fully denoised latent of frame n-1 as the noise prior (after layer normalization) for denoising frame n, so video is generated frame-by-frame online; a streaming data sampler trains on consecutive frames across iterations with a latent cache, keeping training memory constant with video length.
Key results
- Glad achieves FID 12.57 and FVD 207 on nuScenes validation without reference frame, and FID 11.18 and FVD 188 with reference frame, beating MagicDrive (FID 16.20) and Panacea (FID 16.96, FVD 139 on FVD only partially).
- Using generated data to pre-train StreamPETR with nuImage pre-training raises NDS from 49.4 to 51.3 and AMOTA from 33.9 to 36.5.
- Glad reaches 74.9% of Oracle mAP and 83.6% of Oracle NDS when detecting on 8-frame generated validation videos (mAP 28.3, NDS 41.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Glad instead of Drive-WM for online multi-frame generation | nuScenes video generation · nuScenes validation | FID: 15.80 -> 12.57 | Helped |
| Start generation from a reference frame instead of Gaussian noise | Glad · nuScenes validation | FID: 12.57 -> 11.18 | Helped |
| Add Glad-generated data to nuImage and real training data | StreamPETR · nuScenes validation | NDS: 49.4 -> 51.3; +1.9 | Helped |
| Pretrain MapTR on Glad-generated data | MapTR-tiny · nuScenes validation, HD map construction | mAP: +4.9 | Helped |
| Add latent variable propagation, using prior-frame denoised latents as noise prior (ablation) | Glad · nuScenes validation | FID: 20.85 -> 12.57 | Helped |
| Add latent variable propagation (ablation) | Glad · nuScenes validation, generated-data detection | mAP: 26.8 -> 28.3 | Helped |
| Use 2 rather than 1 training video chunk (ablation) | Glad · nuScenes validation | FID: 14.43 -> 12.57 | Helped |
| Increase training video chunks from 2 to 5 (ablation) | Glad · nuScenes validation | FID: 12.57 -> 14.13 | Hurt |
| Generate 16-frame rather than 8-frame videos (ablation) | Glad · nuScenes validation, generated-data detection | mAP: 28.3 -> 27.7 | Hurt |
| Use Glad-generated denoised latents instead of previous-frame latent features directly as noise prior (ablation) | Glad · nuScenes validation | FID: 49.02 -> 12.57 | Helped |
| Use Glad-generated denoised latents instead of Gaussian-noisy previous-frame latents as noise prior (ablation) | Glad · nuScenes validation | FID: 32.43 -> 12.57 | Helped |
| Replace LVP with a temporal attention layer (ablation) | Glad · nuScenes validation | FID: 12.57 -> 17.91 | Hurt |
Surprising findings
- Using a temporal attention layer instead of latent propagation gave a worse FID (17.91 vs 12.57) while costing 66GB memory and 5 days of training versus 38GB and 1 day.
- Propagating noisy (re-noised) latents or raw real-image latents as noise prior causes severe error accumulation, while caching the denoised latent works best.
Limitations
- Struggles to generate high-quality video under highly dynamic scenes, with poor temporal consistency of objects.
- Generated data yields only modest gains; detection on generated data reaches 74.9% of Oracle mAP.
- Arbitrary-length generation is claimed only theoretically; experiments use up to 16 frames.
Experimental setup
Datasets: nuScenes Argoverse 2 CODA nuImage
Benchmarks: nuScenes
Metrics: FID FVD mAP NDS AMOTA AMOTP AP_ped AP_divider AP_boundary
Baselines: BEVGen BEVControl MagicDrive DriveDreamer WoVoGen Panacea DrivingDiffusion Drive-WM
Builds on: Stable Diffusion Panacea ControlNet StreamPETR MapTR
Closest papers by meaning
- SimGen: Simulator-conditioned Driving Scene Generation (NeurIPS 2024 · similarity 0.89)
- DriveScape: High-Resolution Driving Video Generation by Multi-View Feature Fusion (CVPR 2025 · similarity 0.88)
- UniScene: Unified Occupancy-centric Driving Scene Generation (CVPR 2025 · similarity 0.88)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.87)
- Generating Multimodal Driving Scenes via Next-Scene Prediction (CVPR 2025 · similarity 0.87)
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.