Research map / Multi-agent systems and games / LLM agents

RLFTSim: Realistic and Controllable Multi-Agent Traffic Simulation via Reinforcement Learning Fine-Tuning

CVPR 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLFTSim, an on-policy RL fine-tuning of the SMART traffic simulator using a dense leave-one-out version of the WOSAC realism meta-metric (MLOO), plus hindsight-relabeled goal conditioning to distill controllability.

Problem

Learned multi-agent traffic simulators trained with open-loop imitation learning suffer closed-loop distribution shift and unrealistic behavior; the WOSAC realism meta-metric is sparse and sample-inefficient as an RL reward, and controllability (goal-reaching) is limited.

Approach

Use the realism meta-metric as the RL reward via a per-rollout leave-one-out construction (MLOO) that is dense and has lower variance, within REINFORCE with KL regularization; distill goal-conditioning with hindsight goal relabeling and a combined realism-plus-goal reward.

Key results

• RLFTSim achieves RMM 0.7867 on the WOSAC private test split, state-of-the-art over TrajTok (0.7861) and CAT-K (0.7856), above the SMART-tiny reference (0.7824 re-trained).
• On the validation set, MLOO reward yields RMM 0.7830 vs 0.7804 for the reference model, outperforming RLOO-RMM (0.7821) and minADE RLOO (0.7801).
• Indication-based goal conditioning with soft goals reaches a passing miss rate of 9.180 vs 16.631 for goal-free RLFTSim.

Surprising findings

• Heuristic rewards (collision, off-road, minADE) optimize their own objectives but do not improve realism near the optimal stage.
• Goal-conditioned fine-tuning with concatenated goal coordinates reduces realism relative to indication-based goal encoding.

Limitations

• Token-based representation with half-second segments may reduce responsiveness in highly dynamic scenarios.
• Goal-conditioned controllability is reasonable but imperfect; goal-reaching rates have room for improvement.
• RMM is a proxy for realism and may saturate due to metric limitations.

Experimental setup

Datasets: Waymo Open Motion Dataset

Benchmarks: WOSAC (Waymo Open Sim Agents Challenge)

Metrics: RMM kinematic interactive map-based minADE passing miss rate

Baselines: SMART-tiny SMART-tiny CAT-K TrafficBotsV1.5 VBD MVTE Trajeglish KiGRAS DRoPE-Traj GUMP BehaviorGPT UniMM TrajTok

Builds on: SMART REINFORCE Leave-One-Out (RLOO) Hindsight Experience Replay CAT-K WOSAC realism meta-metric

Closest papers by meaning

• Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.93)
• Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models (CVPR 2025 · similarity 0.88)
• RLfOLD: Reinforcement Learning from Online Demonstrations in Urban Autonomous Driving (AAAI 2024 · similarity 0.87)
• IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learning (ICLR 2025 · similarity 0.87)
• Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning (AAAI 2024 · similarity 0.86)
• BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction (NeurIPS 2024 · similarity 0.86)