Research map / Multi-agent systems and games / LLM agents

Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SMART-R1, an R1-style SFT-RFT-SFT post-training pipeline with a metric-oriented policy optimization (MPO) for next-token-prediction traffic simulators, reaching first place on the 2025 WOSAC leaderboard.

Problem

Data-driven NTP traffic simulators are trained with cross-entropy/behavior cloning, which does not directly optimize non-differentiable simulation realism metrics (collisions, off-road rates) and suffers covariate shift in closed-loop rollouts.

Approach

Applies R1-style reinforcement fine-tuning to a motion-token NTP simulator: the Realism Meta score is used as a scalar reward, advantage is reward minus a fixed empirical threshold (no group sampling or critic), and an extra closed-loop SFT is added after RFT to mitigate forgetting.

Key results

• SMART-R1 reaches a Realism Meta score of 0.7858 on the 2025 WOSAC leaderboard, ranking first, with minADE 1.2885.
• On the 2% validation split, the SFT-RFT-SFT pipeline achieves 0.7746 Realism Meta versus 0.7725 for the BC-pretrained model.
• In the RFT policy comparison, MPO (0.7740) outperforms GRPO (0.7701), DPO (0.7682), and PPO (0.7611) on the same SFT baseline setting.

Surprising findings

• PPO, DPO, and GRPO all degrade Realism Meta relative to the one-stage SFT baseline in the authors' experiments.
• Two consecutive SFT phases without an intermediate RFT stage slightly degrade performance relative to a single SFT stage.

Limitations

• Ablations use a 2% validation split rather than the full validation set due to evaluation cost.
• The advantage uses a fixed empirical threshold (alpha=0.77) instead of a learned value or group baseline, which relies on the reward distribution being predictable.

Experimental setup

Datasets: Waymo Open Motion Dataset

Benchmarks: Waymo Open Sim Agents Challenge 2025

Metrics: Realism Meta metric minADE kinematic metrics interactive metrics map-based metrics

Baselines: SMART SimFormer UniMM comBOT RLFTSim CLSFT TrajTok PPO DPO GRPO

Builds on: SMART CAT-K rollouts GRPO DeepSeek-R1 PPO

Closest papers by meaning

• RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuning (CVPR 2026 · similarity 0.93)
• BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction (NeurIPS 2024 · similarity 0.90)
• IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learning (ICLR 2025 · similarity 0.89)
• Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models (CVPR 2025 · similarity 0.89)
• DriveGPT: Scaling Autoregressive Behavior Models for Driving (ICML 2025 · similarity 0.87)