Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Trajeglish: Traffic Modeling as Next-Token Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trajeglish tokenizes WOMD agent trajectories with a data-driven k-disks vocabulary (384 tokens, ~1 cm error) and models multi-agent motion with a GPT-like encoder-decoder that conditions on intra-timestep actions, topping the Waymo Sim Agents Benchmark.
Problem
Simulating realistic multi-agent driving scenarios, seeded from recorded logs, that remain reactive to an autonomous vehicle's actions at every timestep.
Approach
Discretize agent state transitions with k-disks (a Poisson-disk-like sampling of transition templates optimized for corner distance) and model the multi-agent token sequence autoregressively in time and across agents within a timestep, so each agent's action conditions on previously chosen actions of other agents.
Key results
- Trajeglish achieves a realism meta metric of 0.5339 on the WOMD Sim Agents test set, the top score on the leaderboard at submission time, with interactive metrics 0.5811 (9.9% above the prior best).
- k-disks with 384 templates achieves expected corner discretization error of 1.18 cm, versus 6.13 cm for k-means and 14.09 cm for (x,y,h)-grid.
- Fine-tuning the pretrained model on nuScenes reaches a far lower validation NLL than training from scratch, while LoRA underperforms full fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| model intra-timestep interactions between agents | Trajeglish · WOMD Sim Agents Benchmark | interactive metrics: +9.9% | Helped |
| use k-disks instead of k-means for template selection | WOMD tokenization, |V|=384 · WOMD | expected discretization error (cm): 6.13 -> 1.18 | Helped |
| add intra-timestep conditioning (ablation) | Trajeglish · full-control rollouts | collision rate | Helped |
| add noisy tokenization during training (ablation) | Trajeglish · full-control rollouts | rollout performance | Helped |
| remove map encoder (ablation) | Trajeglish · partial-control rollouts | single-agent ADE | Hurt |
| place the controlled agent last so it can condition on replay agents' current-timestep actions (ablation) | Trajeglish · partial-control rollouts | ADE | Helped |
| add a heading smoother (ablation) | Trajeglish · WOMD Sim Agents validation | kinematic metrics: 0.3963 -> 0.4065 | Helped |
| increase sampling temperature from 1.25 to 1.5 and nucleus from 0.995 to 1.0 (ablation) | Trajeglish · 2023 WOMD Sim Agents validation | realism meta metric: 0.5176 -> 0.5312 | Helped |
| fine-tune the pretrained model instead of training from scratch on nuScenes | Trajeglish · nuScenes validation | NLL | Helped |
| use LoRA instead of full-model fine-tuning | Trajeglish · nuScenes validation | NLL | Hurt |
Surprising findings
- Sampling at higher temperature (1.5) improved sim-agent realism and interaction metrics, which the authors attribute to coverage being penalized more strongly than calibration.
- Intra-timestep dependence matters mainly when little context is given; it becomes much less important beyond about 4 timesteps of history.
Limitations
- Kinematic realism lags prior regression-based models; the authors needed a heading smoother to partially close the gap.
- Scaling beyond the ~1B tokens of WOMD is untested; the scaling study is preliminary.
- Map object definitions are arbitrary and may hinder transfer across datasets such as nuScenes.
Experimental setup
Datasets: Waymo Open Motion Dataset nuScenes
Benchmarks: WOMD Sim Agents Benchmark nuScenes
Metrics: realism meta metric kinematic metrics interactive metrics map-based metrics minADE collision rate negative log-likelihood discretization error
Baselines: Constant Velocity Wayformer MTR+++ Joint-Multipath++ MTR_E MVTA MVTE no intra marginal (x,y,h)-grid (x,y)-grid k-means
Builds on: VectorNet Wayformer Trajectory Transformer k-means++ flash attention
Closest papers by meaning
- BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction (NeurIPS 2024 · similarity 0.90)
- Language-Driven Interactive Traffic Trajectory Generation (NeurIPS 2024 · similarity 0.89)
- DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States (NeurIPS 2024 · similarity 0.88)
- Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modeling (NeurIPS 2025 · similarity 0.88)
- Motion Forecasting in Continuous Driving (NeurIPS 2024 · similarity 0.88)
- UniMotion: A Unified Motion Framework for Simulation, Prediction and Planning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.