Research map / Reinforcement learning / Visual and representation RL
Transition-Informed Reinforcement Learning for Large-Scale Stackelberg Mean-Field Games
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TIRL, a model-free RL framework for Stackelberg mean-field games that learns the followers' transition function and uses an instantiated empirical Fokker-Planck equation to generate extra training data, plus regularization for stability, scaling to 100,000 followers on EDRP and MTFG.
Problem
Learning a leader's incentive policy in large-scale Stackelberg mean-field games with many homogeneous followers, where classical methods need exact dynamics, model-free RL is data-inefficient, and leader-follower interaction makes training unstable.
Approach
Learn the followers' transition function with a neural network and use it to instantiate the empirical Fokker-Planck equation so the state distribution evolves directly (non-atomically) without extra environment interaction; add entropy and L1 regularization plus weight clipping to stabilize leader policy learning.
Key results
- On EDRP with N=100,000 followers, Reg-TIRL reaches ORR 0.94±0.00 versus 0.82±0.00 for the best-scaling SMFU-w/o-L baseline, and M3RL/EBPG are infeasible beyond N=1,000.
- TIRL and Reg-TIRL have shorter runtimes than SMFU-type methods as the number of followers grows, scaling to 100,000 followers.
- Regularization (Reg-SMFU, Reg-TIRL) markedly reduces sensitivity to leader policy initialization across random seeds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add transition-informed learning using the learned follower transition function and empirical Fokker-Planck equation | SMFU · EDRP | ORR | Mixed |
| add regularization to TIRL | TIRL · EDRP | ORR | Helped |
| add regularization to SMFU | SMFU · EDRP | ORR | Helped |
| add the leader's incentive (ablation) | SMFU-w/o-L · EDRP | ORR: 0.76±0.00 -> 0.79±0.02 | Helped |
| replace interaction-based follower transitions with transitions computed using the learned model and empirical Fokker-Planck equation | SMFU · EDRP | runtime | Helped |
Surprising findings
- Plain TIRL is sometimes less stable than SMFU at large N, and regularization is needed to match or exceed it (e.g., N=50,000 Reg-TIRL outperforms TIRL).
Limitations
- Only a single leader and homogeneous followers are considered.
- Relies on a learned transition approximator, and equivalence of the EFP to the true FP under parameterized policies is not rigorously proven.
- Evaluated only in two simulated environments (EDRP and MTFG).
Experimental setup
Datasets: NYC taxi trip records (Manhattan pickups/drop-offs) simulated MTFG food gathering environment
Benchmarks: EDRP (e-hailing driver re-positioning) MTFG (multiple-type food gathering)
Metrics: order response rate (ORR) collection ratio (CR) runtime (h)
Baselines: M3RL EBPG SMFU-w/o-L SMFU Reg-SMFU TIRL
Builds on: Stackelberg mean-field game Fokker-Planck equation MBPO M3RL EBPG
Closest papers by meaning
- Learning in Stackelberg Mean Field Games: A Non-Asymptotic Analysis (NeurIPS 2025 · similarity 0.92)
- Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics (NeurIPS 2025 · similarity 0.88)
- Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games (NeurIPS 2026 · similarity 0.88)
- Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation (ICLR 2025 · similarity 0.88)
- Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL (ICML 2024 · similarity 0.87)
- Learning Discrete-Time Major-Minor Mean Field Games (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.