Research map / Reinforcement learning / Imitation learning
ReSim: Reliable World Simulation for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds ReSim, a controllable driving world model trained on real driving videos plus CARLA non-expert (hazardous) data, with a Video2Reward model that scores simulated futures for planning and policy selection.
Problem
Driving world models trained only on safe expert human data hallucinate or fail to follow hazardous non-expert actions, limiting their use for policy evaluation and reward estimation.
Approach
Enrich real-world driving logs (OpenDV, NAVSIM) with CARLA non-expert and expert data, train a CogVideoX-based diffusion transformer with trajectory conditioning, unbalanced noise sampling and a dynamics consistency loss, and derive rewards from simulated videos with a DINOv2-based Video2Reward head trained on CARLA infraction scores.
Key results
- On NAVSIM navtest, the video prediction-based policy reaches PDMS 86.6, versus 84.6 for LAW and 84.0 for Transfuser.
- On nuScenes without action conditioning, ReSim reaches FID 5.2 and FVD 50.4 zero-shot, versus 6.9 and 89.4 for Vista.
- In reward-guided policy selection on 300 hard NAVSIM scenarios, ReSim with Video2Reward reaches PDMS 74.1, versus 66.8 for uniform ensemble and 59.2 for the Vista reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReSim instead of Vista for action-free prediction | Driving world models · Waymo Open validation | Trajectory Difference: 80% | Helped |
| Use ReSim instead of Vista for expert-action conditioning | Driving world models · Waymo Open validation | Trajectory Difference: 54% | Helped |
| Add simulated CARLA data to training (ablation) | ReSim · Waymo Open validation | Trajectory Difference | Helped |
| Add simulated CARLA data to training (ablation) | ReSim · Open-world non-expert action controllability | Trajectory following | Helped |
| Add simulated CARLA data to ReSim training (ablation) | ReSim + Video2Reward · NAVSIM | Reward-guided policy selection PDMS | Helped |
| Use unbalanced rather than uniform noise sampling (ablation) | ReSim · Driving video prediction | Motion and scenario consistency | Helped |
| Add dynamics consistency loss (ablation) | ReSim · Driving video prediction | Future prediction coherence | Helped |
| Set maximum latent-motion interval to K=4 (ablation) | ReSim with dynamics consistency loss · Driving video prediction | Future prediction coherence | Helped |
| Use ReSim with Video2Reward for candidate trajectory selection | Transfuser and LTF candidates · NAVSIM subset, 300 challenging scenarios | PDMS: 47.7 and 47.2 -> 74.1 | Helped |
Limitations
- Inference is slow due to iterative denoising (about two minutes per 4s video on one A100).
- Trained on front-view video only, so multi-view planners such as UniAD and VAD cannot be directly applied in the simulator.
- Hallucinations and visual artifacts remain in multi-agent scenes and for occluded objects.
Experimental setup
Datasets: OpenDV NAVSIM CARLA Waymo Open nuScenes Bench2Drive
Benchmarks: NAVSIM navtest nuScenes validation Waymo validation
Metrics: PDMS FID FVD Trajectory Difference reward correlation human preference
Baselines: Vista DriveGAN DriveDreamer DriveDreamer-2 WoVoGen Drive-WM GenAD GEM UniAD Transfuser DrivingGPT LAW
Builds on: CogVideoX Vista DINOv2 NAVSIM OpenDV
Closest papers by meaning
- NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (NeurIPS 2024 · similarity 0.89)
- DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model (NeurIPS 2024 · similarity 0.88)
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (NeurIPS 2024 · similarity 0.88)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.88)
- DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation (CVPR 2025 · similarity 0.87)
- RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.