Research map / Label-efficient and robust learning / Recognition and long-tail
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NAVSIM, a non-reactive, BEV-abstraction simulation benchmark on the OpenScene/nuPlan data with a PDM Score metric that aligns better with closed-loop evaluation than displacement errors, and reports results from a 2024 CVPR challenge.
Problem
Benchmarking vision-based end-to-end driving policies is hard: open-loop displacement-error evaluation does not reflect closed-loop performance, while closed-loop simulators are costly and have a sensor domain gap to real data.
Approach
Evaluate sensor-based driving policies on real data by unrolling a non-reactive BEV simulation for a 4 s horizon after querying the agent only once, yielding simulation-based metrics (PDMS: NC, DAC, TTC, comfort, EP) that are computable open-loop and correlate better with closed-loop scores than ADE-style metrics.
Key results
- PDMS correlates better with closed-loop score than open-loop score (OLS) across planner types; PDMS is positively correlated with CLS for every planner type.
- On navtest, TransFuser (PDMS 84.0) and PARA-Drive (84.0) match or slightly exceed UniAD (83.4), while a constant-velocity baseline scores 20.6 and the human reaches 94.8.
- The NAVSIM challenge received 463 submissions from 143 teams; the winning entry extended TransFuser with trajectory sampling and learned subscore scoring.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add velocity and acceleration to goal-only ego status (ablation) | TransFuser · navtest | PDMS: 81.8 -> 84.0 | Helped |
| Add acceleration to goal and velocity ego status (ablation) | TransFuser · navtest | PDMS: 82.3 -> 84.0 | Helped |
| Expand camera FOV from 60° to the default 140° (ablation) | TransFuser · navtest | PDMS: 80.3 -> 84.0 | Helped |
| Expand camera FOV to 160° (3 cameras) (ablation) | TransFuser · navtest | PDMS: 84.0 -> 82.8 | Hurt |
| Expand camera FOV to 240° (5 cameras) (ablation) | TransFuser · navtest | PDMS: 84.0 -> 84.1 | No clear effect |
| Use default LiDAR range instead of restricting it to 16m in all directions (ablation) | TransFuser · navtest | PDMS: 79.1 -> 84.0 | Helped |
| Extend LiDAR range to F:64, B:32, L:32, R:32 (ablation) | TransFuser · navtest | PDMS: 84.0 -> 84.3 | No clear effect |
| Extend LiDAR range to 64m in all directions (ablation) | TransFuser · navtest | PDMS: 84.0 -> 81.0 | Hurt |
| Add BEV segmentation supervision (ablation) | TransFuser · navtest | PDMS: 81.6 -> 84.0 | Helped |
| Add 3D detection supervision (ablation) | TransFuser · navtest | PDMS: 84.0 -> 84.0 | No clear effect |
| Use PDMS instead of OLS for open-loop evaluation | NAVSIM planner evaluation · navmini | Correlation with CLS | Helped |
Surprising findings
- Simple TransFuser and camera-only Latent TransFuser match recent large-scale architectures UniAD and PARA-Drive despite far lower training compute.
- An ego-status-only MLP reaches a PDMS of 65.6, indicating a clear gap to sensor-based agents unlike on nuScenes.
Limitations
- Non-reactive simulation over a short 4 s horizon ignores agent reactiveness and compounding closed-loop errors.
- Rear-end collisions into the ego vehicle are not counted as at-fault, and traffic-light or stop-sign compliance is not evaluated.
- Dataset noise and label-space gaps in nuPlan/OpenScene (e.g., camera parameter errors, missing classes) persist.
Experimental setup
Datasets: OpenScene nuPlan navtrain navmini nuScenes
Benchmarks: navtest navmini CVPR 2024 NAVSIM Challenge private test NAVSIM 1.1 leaderboard
Metrics: PDM Score no at-fault collision drivable area compliance time-to-collision comfort ego progress open-loop score closed-loop score Spearman correlation Pearson correlation
Baselines: Constant Velocity Ego Status MLP TransFuser Latent TransFuser UniAD PARA-Drive Hydra-MDP
Builds on: nuPlan OpenScene PDM-Closed TransFuser nuScenes
Closest papers by meaning
- Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving (NeurIPS 2024 · similarity 0.89)
- ReSim: Reliable World Simulation for Autonomous Driving (NeurIPS 2025 · similarity 0.89)
- DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation (CVPR 2025 · similarity 0.87)
- DiffScene: Diffusion-Based Safety-Critical Scenario Generation for Autonomous Vehicles (AAAI 2025 · similarity 0.86)
- Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution (NeurIPS 2025 · similarity 0.86)
- SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.