Research map / Multi-agent systems and games / Human-AI and goal-conditioned
BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BehaviorGPT, a fully autoregressive homogeneous Transformer with next-patch prediction (NP3) for multi-agent traffic simulation, reaching first place in the 2024 Waymo Open Sim Agents Challenge with 3M parameters.
Problem
Data-driven agent simulation for autonomous driving relies on heterogeneous encoder-decoder architectures that separate history and future, which is complex and data-inefficient; autoregressive next-step prediction also suffers from compounding errors and trivial copy-the-current-state shortcuts.
Approach
Treat every time step as the "current" one in a decoder-only Transformer with triple attention (temporal, agent-map, agent-agent) over trajectory patches, and train each patch to predict the next multi-step patch (NP3) instead of the next single state, using relative spacetime representations from QCNet.
Key results
- On the WOSAC 2024 test set, BehaviorGPT with 3M parameters reaches REALISM 0.7473 and minADE 1.4147, winning first place.
- Patch size 10 (1-second patches) outperforms patch size 1 on the validation set (minADE 1.5203 vs 2.3752; REALISM 0.7335 vs 0.6783).
- Removing agent-agent self-attention drops validation REALISM from 0.7349 to 0.6659 and raises minADE from 1.6247 to 2.1489.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase patch size from 1 to 10 time steps (ablation) | BehaviorGPT on WOMD validation · WOMD validation | REALISM: 0.6783 -> 0.7335 | Helped |
| increase patch size from 5 to 10 time steps (ablation) | BehaviorGPT on WOMD validation · WOMD validation | minADE: 1.5599 -> 1.5203 | Helped |
| increase replan frequency from 1 Hz to 2 Hz (ablation) | BehaviorGPT, patch size 10 · WOMD test | REALISM: 0.7414 -> 0.7473 | Helped |
| increase replan frequency from 2 Hz to 5 Hz (ablation) | BehaviorGPT, patch size 10 · WOMD test | REALISM: 0.7473 -> 0.7342 | Hurt |
| add agent-agent self-attention (ablation) | BehaviorGPT · WOMD validation | minADE: 2.1489 -> 1.6247 | Helped |
| train on 50% rather than 20% of the data (ablation) | BehaviorGPT with 4 decoding blocks · WOMD validation | minADE: 1.4881 -> 1.4060 | Helped |
| train on 100% rather than 50% of the data (ablation) | BehaviorGPT with 4 decoding blocks · WOMD validation | minADE: 1.4060 -> 1.3804 | Helped |
| increase model depth from 2 to 4 decoding blocks (ablation) | BehaviorGPT, hidden size 128 · WOMD validation | REALISM: 0.7349 -> 0.7396 | Helped |
| increase model width from 64 to 192 (ablation) | BehaviorGPT, 2 decoding blocks · WOMD validation | minADE: 1.9637 -> 1.4993 | Helped |
| train on 5.0-second sequences and infer on 9.1-second sequences instead of matching 9.1-second training and inference (ablation) | BehaviorGPT · WOMD, inference 9.1 sec | REALISM: 0.7349 -> 0.7333 | No clear effect |
| use BehaviorGPT instead of GUMP | WOSAC test · WOSAC test | REALISM: 0.7431 -> 0.7473 | Helped |
| use BehaviorGPT instead of GUMP | WOSAC test · WOSAC test | ANG ACCEL: 0.6353 -> 0.5544 | Hurt |
Surprising findings
- Increasing replan frequency from 1 Hz to 2 Hz improved performance, but 5 Hz degraded it, indicating a moderate replan rate matters for temporal stability.
- A model trained on 5-second sequences generalized to 9.1-second inference with performance close to the model trained on full length.
Limitations
- Kinematic performance (linear/angular speed and acceleration) is weaker than some baselines.
- No support for controllable generation via language or goal prompts.
- Autoregressive models still accumulate errors leading to offroad trajectories.
Experimental setup
Datasets: Waymo Open Motion Dataset
Benchmarks: Waymo Open Sim Agents Challenge (WOSAC) 2024
Metrics: minADE REALISM linear speed linear accel angular speed angular accel dist to object collision TTC dist to road edge
Baselines: Linear Extrapolation TrafficBotsV1.5 VBD MVTE GUMP
Builds on: QCNet GPT Trajeglish MotionLM
Closest papers by meaning
- DriveGPT: Scaling Autoregressive Behavior Models for Driving (ICML 2025 · similarity 0.92)
- Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.90)
- Trajeglish: Traffic Modeling as Next-Token Prediction (ICLR 2024 · similarity 0.90)
- SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction (NeurIPS 2024 · similarity 0.90)
- Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving (NeurIPS 2026 · similarity 0.88)
- UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.