Research map / Architectures and efficiency / RNNs and normalization
DriveGPT: Scaling Autoregressive Behavior Models for Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales an LLM-style autoregressive transformer behavior model for driving to 1.4B parameters on 120M human driving segments, studying data, model, and compute scaling for planning and WOMD motion prediction.
Problem
Whether LLM-like scaling trends in data, parameters, and compute carry over to autonomous driving behavior modeling, where prior behavior models were limited to a few million training examples and small parameter counts.
Approach
Models driving as sequential decision-making: a transformer encoder summarizes the scene and an LLM-style autoregressive decoder predicts discrete Verlet-acceleration actions one step at a time, scaled by orders of magnitude in data and parameters.
Key results
- On WOMD test, DriveGPT-Finetune achieves minADE 0.5240 and minFDE 1.0538, best among compared methods without ensembling, though its soft mAP (0.3857) trails ControlMTR and MotionLM.
- Data scaling fits log(L) = -0.102 log(D) + 2.663 (R^2 0.986); a 120M-trained 26M model cuts normalized collision rate to 0.196 from 1.000 at 2.2M.
- Model scaling improves driving metrics up to 94M parameters on the 120M dataset, with collision rate still improving at 163M, and an 8M model runs closed-loop under 50ms on a single onboard GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase training data from 2.2M to 120M samples | DriveGPT, 26M parameters · Internal AV planning test | normalized mADE: 1.000 -> 0.489 | Helped |
| Increase training data from 2.2M to 120M samples | DriveGPT, 26M parameters · Internal AV planning test | normalized collision rate: 1.000 -> 0.196 | Helped |
| Increase model size from 8M to 94M parameters | DriveGPT, 120M data · Internal AV planning test | normalized mADE: 1.000 -> 0.937 | Helped |
| Increase model size from 8M to 163M parameters | DriveGPT, 120M data · Internal AV planning test | normalized collision rate: 1.000 -> 0.817 | Helped |
| Pretrain on the 120M internal dataset before WOMD finetuning | DriveGPT · WOMD test | minADE: 0.5279 -> 0.5240 | Helped |
| Pretrain on the 120M internal dataset before WOMD finetuning | DriveGPT · WOMD test | Soft mAP: 0.3795 -> 0.3857 | Helped |
| Use autoregressive rather than one-shot decoding (ablation) | DriveGPT decoder · Motion prediction, minFDE at 6 seconds | minFDE | Mixed |
| Scale only the decoder instead of scaling both encoder and decoder (ablation) | DriveGPT · Internal validation | validation loss | Hurt |
| Scale only the encoder instead of scaling both encoder and decoder (ablation) | DriveGPT · Internal validation | validation loss | Hurt |
| Vary attention-head count while keeping hidden dimension fixed (ablation) | DriveGPT transformers · Internal validation | validation loss | No clear effect |
| Increase internal pretraining data from 40M to 120M samples (ablation) | DriveGPT-Finetune · WOMD vehicle validation | mADE: 0.6368 -> 0.6310 | Helped |
| Increase internal pretraining data from 80M to 120M samples (ablation) | DriveGPT-Finetune · WOMD vehicle validation | mADE: 0.6341 -> 0.6310 | Helped |
Surprising findings
- Soft mAP is weaker than baselines because compounded autoregressive action probabilities are noisy, and the gap grows with prediction horizon.
- Scaling only the encoder has little impact beyond 10M parameters, and the one-shot decoder was harder to scale than the autoregressive decoder.
Limitations
- Data scaling experiments use a proprietary internal dataset that is not released, limiting reproducibility.
- Autoregressive decoder yields suboptimal probability estimates, reducing soft mAP.
- Gains saturate with model size beyond 94M, and larger datasets were left to future work.
Experimental setup
Datasets: internal research driving dataset (120M segments) Waymo Open Motion Dataset (WOMD)
Benchmarks: Waymo Open Motion Dataset motion prediction internal AV planning test set
Metrics: validation loss minADE minFDE miss rate soft mAP offroad rate collision rate
Baselines: MTR HDGT HPTR ControlMTR MTR++ Wayformer MotionLM one-shot decoder VectorNet PlanTF QCNet GUMP
Builds on: MTR LLM scaling laws Wayformer
Closest papers by meaning
- BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction (NeurIPS 2024 · similarity 0.92)
- Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving (NeurIPS 2026 · similarity 0.91)
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.88)
- BAT: Behavior-Aware Human-Like Trajectory Prediction for Autonomous Driving (AAAI 2024 · similarity 0.88)
- UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning (NeurIPS 2025 · similarity 0.88)
- AlignDrive: Aligned Lateral-Longitudinal Planning for End-to-End Autonomous Driving (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.