Research map / Reinforcement learning / Visual and representation RL

Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: RL for combinatorial routing · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BehaviorBench, a closed-loop driving benchmark for PufferDrive with a nuPlan interface, two 1k WOMD scenario splits (Interactive1k, Random1k), a diverse set of reactive traffic agents (IDM, PPO, SMART, reward-conditioned styles), and a PDM+PPO hybrid planner.

Problem

Standard planning benchmarks (nuPlan Val14) are too easy and use a single IDM traffic model, so large-scale RL driving policies trained via self-play cannot be fairly evaluated and their generalization to other traffic behaviors is unknown.

Approach

A unified evaluation suite bridging PufferDrive and nuPlan, with interaction-scored WOMD splits and heterogeneous reactive traffic agents (including reward-conditioned aggressive/normal/cautious PPO agents), plus a hybrid PDM+PPO planner that falls back to PPO when all PDM proposals score zero.

Key results

• On Interactive1k, the PDM+PPO hybrid scores 56.71 averaged across traffic agents, versus 51.46 for PPO and 24.55 for PDM.
• PPO trained via self-play achieves near-zero at-fault collision against PPO traffic but an order of magnitude higher against IDM traffic, with no monotonic improvement over 10^11 agent steps.
• On interPlan, PPO scores 42.10 versus PDM's 41.68, while on nuPlan Val14 PDM dominates PPO (92.13 vs 67.24).

Surprising findings

• Self-play PPO policies generalize poorly to other traffic agents: collision rate stays high against IDM despite training scale, so scaling agent steps does not close the gap.
• PPO on Val14 scores 67.24 vs PDM 92.13, but on interPlan PPO slightly beats PDM, reversing the ranking.

Limitations

• Traffic lights and stop signs are not integrated; agents are removed on goal completion as a workaround.
• Reward-conditioned traffic agents occasionally deviate from idealized behavior.
• Simulator assumes full observability; pedestrians and cyclists are non-controllable log replay.

Experimental setup

Datasets: Waymo Open Motion Dataset (WOMD) validation set nuPlan Val14 interPlan WOSAC 2024

Benchmarks: BehaviorBench Interactive1k BehaviorBench Random1k nuPlan Val14 interPlan WOSAC 2024

Metrics: benchmark score at-fault collision rate off-road rate goal completion comfort lane alignment lane center alignment WOSAC realism meta score minADE ADE

Baselines: IDM PDM PPO SMART DiffusionPlanner

Builds on: PufferDrive GigaFlow PDM-Closed Intelligent Driver Model PPO SMART nuPlan interPlan

Closest papers by meaning

• DriveGPT: Scaling Autoregressive Behavior Models for Driving (ICML 2025 · similarity 0.91)
• IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learning (ICLR 2025 · similarity 0.89)
• Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.89)
• Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.88)
• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.88)
• Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving (NeurIPS 2024 · similarity 0.88)