Research map / Multi-agent systems and games / Human-AI and goal-conditioned
SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
SMART tokenizes vectorized road maps and agent trajectories into discrete tokens and trains a GPT-style decoder-only transformer with next-token prediction for scalable, zero-shot multi-agent driving motion generation.
Problem
Data-driven autonomous driving motion generation is limited by dataset size and domain gaps between datasets, and existing regression-based or encoder-decoder NTP models generalize poorly and scale poorly for real-time simulation.
Approach
Discretize agent trajectories (k-disks motion vocabulary with noised tokenization) and road vectors into tokens, then model motion with a factorized decoder-only transformer that autoregressively predicts next motion tokens conditioned on map tokens, avoiding re-encoding history at each step.
Key results
- SMART 101M achieves a realism meta metric of 0.7614 on WOMD 2024 Sim Agents, the top score among the compared methods.
- SMART trained only on NuPlan (SMART*) reaches 0.72 on the WOSAC 2024 Sim Agents challenge via zero-shot evaluation on WOMD (SMART-zeroshot in the 2024 table).
- Test loss follows a power law in model size with exponent β=-0.157; single-frame inference averages under 10 ms for the 1M-7M models, rising to 46.58 ms for the 101M model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SMART 7M rather than Trajeglish | WOMD 2023 Sim Agents · WOMD 2023 Sim Agents | Realism Meta metric: 0.6451 -> 0.6587 | Helped |
| Scale SMART from 7M to 101M parameters | SMART · WOMD 2024 Sim Agents | Realism Meta metric: 0.7591 -> 0.7614 | Helped |
| Train on NuPlan only and evaluate zero-shot on WOMD | SMART · WOMD 2024 Sim Agents | Realism Meta metric: 0.7591 -> 0.7210 | Hurt |
| Add road vector tokenization (ablation) | SMART, M1 to M2 · WOMD validation, trained on WOMD versus NuPlan | interactive: WOMD: 0.827 -> 0.807; NuPlan: 0.593 -> 0.696 | Mixed |
| Add noised agent tokenization (ablation) | SMART, M2 to M3 · WOMD validation, trained on WOMD versus NuPlan | interactive: WOMD: 0.807 -> 0.809; NuPlan: 0.696 -> 0.750 | Helped |
| Add noised road vector tokenization (ablation) | SMART, M3 to M4 · WOMD validation, trained on WOMD versus NuPlan | interactive: WOMD: 0.809 -> 0.801; NuPlan: 0.750 -> 0.747 | Mixed |
| Add road vector next token prediction (ablation) | SMART, M3 to M5 · WOMD validation, trained on WOMD versus NuPlan | interactive: WOMD: 0.809 -> 0.813; NuPlan: 0.750 -> 0.780 | Helped |
| Add noised road vector tokenization (ablation) | SMART, M5 to M6 · WOMD validation, trained on WOMD versus NuPlan | interactive: WOMD: 0.813 -> 0.803; NuPlan: 0.780 -> 0.785 | Mixed |
| Replace k-disks with VQ-VAE tokenizer (ablation) | SMART 7M · WOMD validation, trained on WOMD | kinematics: 0.453 -> 0.461 | Helped |
| Replace k-disks with VQ-VAE tokenizer (ablation) | SMART 7M · WOMD validation, trained on NuPlan | interactive: 0.785 -> 0.687 | Hurt |
| Increase model scale from 1M to 101M parameters (ablation) | SMART · WOMD validation, multi-dataset training | interactive metrics: 0.782 -> 0.819 | Helped |
| Increase model scale from 1M to 101M parameters (ablation) | SMART · WOMD validation, multi-dataset training | kinematic metrics: 0.423 -> 0.457 | Helped |
Surprising findings
- Adding road vector tokenization reduced overall metrics when training on WOMD alone, which the authors attribute to loss of fine-grained geometric information in discretized map tokens.
- A VQ-VAE learned motion vocabulary performed slightly better in-domain but generalized much worse across datasets than the k-disks vocabulary.
Limitations
- Simple discrete token vocabulary; the authors note an advanced tokenizer could improve performance.
- Scaling experiments limited to models up to 100M parameters by dataset size.
- Not yet validated on motion planning or prediction tasks.
Experimental setup
Datasets: Waymo Open Motion Dataset NuPlan proprietary driving dataset
Benchmarks: WOMD Sim Agents 2023 WOMD Sim Agents 2024 (WOSAC) WOMD validation set
Metrics: realism meta metric kinematic metrics interactive metrics map-based metrics minADE cross-entropy test loss inference time
Baselines: Trajeglish MVTE VPD-PRIOR QCNeXt MultiPath BehaviorGPT GUMP VBD TrafficBOTv1.5 MotionDiffuser
Builds on: MotionLM Trajeglish MTR VQ-VAE
Closest papers by meaning
- BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction (NeurIPS 2024 · similarity 0.90)
- SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction (ICLR 2025 · similarity 0.89)
- SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction (CVPR 2024 · similarity 0.88)
- UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning (NeurIPS 2025 · similarity 0.87)
- Trajeglish: Traffic Modeling as Next-Token Prediction (ICLR 2024 · similarity 0.87)
- Motion Forecasting in Continuous Driving (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.