Research map / Reinforcement learning / RL from feedback
DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DriveDPO, which distills a unified policy distribution from human imitation similarity and rule-based PDM safety scores over an anchor vocabulary, then refines it with iterative Safety DPO using imitation-based rejected-trajectory selection, reaching 90.0 PDMS on NAVSIM.
Problem
End-to-end driving policies trained by imitation learning produce trajectories that look human-like but are unsafe, while existing score-based distillation methods regress per-anchor rule scores and do not optimize the policy distribution directly.
Approach
Fuse imitation similarity and simulator-derived PDM scores into one soft target over anchor trajectories for policy distillation, then apply trajectory-level DPO where chosen is the top-scored anchor and rejected is a low-PDMS anchor closest to the human trajectory.
Key results
- Achieves 90.0 PDMS on NAVSIM Navtest with ResNet-34, surpassing DiffusionDrive (88.1) and WOTE (88.0).
- Unified policy distillation without DPO reaches 88.8 PDMS; full DPO adds 1.2 PDMS.
- On Bench2Drive, reports 30.62% success rate and 62.02 driving score, above TCP (30.00, 59.90).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add unified policy distillation in place of independent per-anchor score regression (ablation) | DriveDPO · NAVSIM Navtest | PDMS: 87.3 -> 88.8 | Helped |
| Add rule-based safety supervision to imitation supervision (ablation) | DriveDPO unified policy distillation · NAVSIM Navtest | PDMS: 82.5 -> 88.8 | Helped |
| Add human supervision to rule-based supervision (ablation) | DriveDPO unified policy distillation · NAVSIM Navtest | NC: 95.3 -> 97.9 | Helped |
| Add DPO fine-tuning (ablation) | DriveDPO · NAVSIM Navtest | PDMS: 88.8 -> 90.0 | Helped |
| Use vanilla preference selection rather than no DPO (ablation) | DriveDPO · NAVSIM Navtest | PDMS: 88.8 -> 89.3 | Helped |
| Use Distance-Based rejected trajectory selection instead of vanilla selection (ablation) | DriveDPO DPO · NAVSIM Navtest | PDMS: +0.9 | Helped |
| Use Imitation-Based rejected trajectory selection instead of vanilla selection (ablation) | DriveDPO DPO · NAVSIM Navtest | PDMS: +1.2 | Helped |
| Use Imitation-Based instead of Distance-Based rejected trajectory selection (ablation) | DriveDPO DPO · NAVSIM Navtest | PDMS: 89.7 -> 90.0 | Helped |
| Apply DriveDPO instead of DiffusionDrive | end-to-end driving policy · NAVSIM Navtest | PDMS: 88.1 -> 90.0 | Helped |
| Apply DriveDPO instead of WOTE | end-to-end driving policy · NAVSIM Navtest | PDMS: 88.0 -> 90.0 | Helped |
| Apply DriveDPO instead of TCP | end-to-end driving policy · Bench2Drive | Driving Score: 59.90 -> 62.02 | Helped |
| Apply DriveDPO instead of VAD | end-to-end driving policy · Bench2Drive | Comfortness: 46.01 -> 26.79 | Hurt |
Surprising findings
- Rule-only supervision without human imitation yields lower PDMS (87.2) and weak NC/TTC, showing human signal is needed alongside rule scores.
Limitations
- Relies on PDMS as the safety criterion, which is a fixed weighted composite that may miss risks.
- Rule-based supervision depends on high-fidelity simulators, which are scarce and limit data scale and diversity.
- Chosen trajectory selection and threshold tau=0.3 are hand-set; Bench2Drive results are only reported against a few baselines.
Experimental setup
Datasets: NAVSIM Navtrain Bench2Drive base training subset
Benchmarks: NAVSIM Navtest Bench2Drive
Metrics: PDMS NC DAC EP TTC Comfort Driving Score Success Rate Efficiency
Baselines: Ego Status MLP VADv2 UniAD LTF Transfuser PARA-Drive LAW DRAMA GoalFlow ARTEMIS DiffusionDrive Hydra-MDP
Builds on: Direct Preference Optimization VADv2 Transfuser Hydra-MDP NAVSIM
Closest papers by meaning
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.89)
- Distill to Think, Foresee to Act: Cognitive-Physical Reinforcement Learning for Autonomous Driving (NeurIPS 2026 · similarity 0.88)
- SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.88)
- RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving (NeurIPS 2026 · similarity 0.87)
- Model-Based Policy Adaptation for Closed-Loop End-to-end Autonomous Driving (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.