Research map / Video / Video segmentation and tracking
Generalized Predictive Model for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds OpenDV-2K, a 2000+ hour text-paired driving video dataset from YouTube and public datasets, and trains GenAD, a latent-diffusion video prediction model with temporal reasoning blocks that generalizes zero-shot and extends to action-conditioned prediction and planning.
Problem
Video prediction models for autonomous driving have been confined to small or restricted datasets and fail to generalize across geographies, sensors, and scenarios.
Approach
Scale driving video pretraining with web-sourced YouTube videos paired with BLIP-2/GPT/classifier-generated text, and adapt SDXL into a video predictor by inserting causal temporal attention and decoupled spatial attention blocks trained with frozen spatial weights.
Key results
- GenAD trained on OpenDV-2K reaches FVD 184 on nuScenes, a 44.5% reduction versus DrivingDiffusion (332), without 3D layout inputs.
- Action-conditioned GenAD-act reduces Action Prediction Error by 20.4% (2.54 to 2.02) versus text-only GenAD.
- A lightweight MLP planner on frozen GenAD reaches ADE 1.23 and FDE 2.31 on nuScenes with 0.8M trainable parameters, versus UniAD's 58.8M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train GenAD on OpenDV-2K rather than nuScenes only | GenAD · nuScenes | FVD: 244 -> 184 | Helped |
| Use GenAD rather than DrivingDiffusion | nuScenes video prediction · nuScenes | FVD: 332 -> 184 | Helped |
| Add deep interaction between temporal and spatial blocks (ablation) | GenAD · OpenDV-YouTube | FVD: 244.44 -> 201.69 | Helped |
| Add temporal causality (ablation) | GenAD with deep interaction · OpenDV-YouTube | CLIPSIM: 0.8409 -> 0.8550 | Helped |
| Add decoupled spatial attention (ablation) | GenAD with temporal causality · OpenDV-YouTube | CLIPSIM: 0.8550 -> 0.8652 | Helped |
| Add future trajectory as an additional condition | GenAD action-conditioned prediction · nuScenes | Action Prediction Error: 2.54 -> 2.02 | Helped |
Surprising findings
- GenAD-nus trained only on nuScenes degrades to nuScenes visual patterns when generating Waymo videos, showing data scale is needed for cross-dataset generalization.
Limitations
- Increased model capacity hurts training efficiency and real-time deployment.
- Planning evaluated only in open-loop on nuScenes with front-view images.
- Video annotations rely on automatic VLM captions (BLIP-2) and classifier commands, with keyword-based manual filtering.
Experimental setup
Datasets: OpenDV-2K nuScenes Waymo KITTI Cityscapes Honda-HDD-Action
Benchmarks: nuScenes Waymo KITTI Cityscapes YouTube validation split
Metrics: FID FVD CLIPSIM Action Prediction Error ADE FDE
Baselines: I2VGen-XL VideoCrafter1 DMVFN DriveGAN DriveDreamer DrivingDiffusion ST-P3 UniAD
Builds on: SDXL latent diffusion models Stable Video Diffusion-style temporal attention UniAD DriveGAN
Closest papers by meaning
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.87)
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024 · similarity 0.86)
- Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving (NeurIPS 2024 · similarity 0.86)
- Glad: A Streaming Scene Generator for Autonomous Driving (ICLR 2025 · similarity 0.86)
- SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction (ICLR 2025 · similarity 0.86)
- DriveGPT: Scaling Autoregressive Behavior Models for Driving (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.