Research map / Multi-agent systems and games / Games and equilibria
Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEDA-FP, a deep RL fictitious-play solver for non-stationary continuous-space mean field games that learns the Nash policy via supervised averaging and the time-dependent population distribution via a time-conditioned neural spline normalizing flow.
Problem
Solving non-stationary mean field games with continuous state and action spaces, including local dependence on the population density, where prior RL methods are restricted to finite spaces, stationary models, or LQ settings.
Approach
Combine fictitious play with DRL (SAC/PPO) for best responses, supervised learning (NLL on a replay buffer) to represent the average policy, and a time-conditioned conditional normalizing flow to model the mean-field flow so it can be sampled and its density evaluated at any time.
Key results
- On the 4-rooms exploration case study, DEDA-FP samples 5000 trajectories more than 10x faster than the benchmark algorithms (text states >10x efficiency).
- DEDA-FP attains exploitability comparable to benchmarks on beach bar and LQ experiments while giving a smoother NE distribution concentrated around the center.
- Theorem gives an epsilon-Nash convergence bound as a sum of best-response, average-policy (supervised learning), and CNF distribution errors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a Conditional Normalizing Flow for the mean-field distribution | DEDA-FP · 4-rooms exploration | sampling speedup: >10× | Helped |
| use a Conditional Normalizing Flow for the mean-field distribution | DEDA-FP · 4-rooms exploration, fixed-time budget of 10s | agents sampled: 10 times more | Helped |
| use a Conditional Normalizing Flow for the mean-field distribution | DEDA-FP · Beach Bar Problem | samples used for distribution approximation: 6x times | Helped |
| learn the average policy with supervised learning | DEDA-FP · Beach Bar Problem and LQ model | exploitability | No clear effect |
| use a Conditional Normalizing Flow to represent the mean-field distribution | DEDA-FP · 4-rooms exploration | final performance | No clear effect |
| use a learning rate of 3 × 10⁻⁴ (ablation) | Deep RL in the center environment · center environment | training stability and convergence speed: 3 × 10⁻⁴ | Helped |
Limitations
- Lacks complete theoretical understanding of deep network training; convergence analysis relies on assumed existence and uniqueness of MFNE.
- Evaluation relies on approximate exploitability that depends on the environment approximation.
- Limited to standard MFGs; extensions to multiple populations, graphon games, and common noise left for future work.
Experimental setup
Datasets: simulated trajectories from beach bar, LQ, 4-rooms, and market model MFG environments
Benchmarks: Beach Bar (continuous) Linear-Quadratic MFG 4-rooms exploration Market model (price impact)
Metrics: approximate exploitability sampling time for 5000 trajectories NE distribution heatmaps
Baselines: Simple Approach (FP with DRL best responses) Learning the NE Policy (Algo 2) Deep FP-style baselines for stationary MFGs
Builds on: Fictitious Play Neural Fictitious Self-Play Neural Spline Flows Soft Actor-Critic Proximal Policy Optimization
Closest papers by meaning
- Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games (NeurIPS 2026 · similarity 0.90)
- Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation (ICLR 2025 · similarity 0.89)
- Learning Discrete-Time Major-Minor Mean Field Games (AAAI 2024 · similarity 0.89)
- Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean Field Games (ICML 2025 · similarity 0.89)
- Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL (ICML 2024 · similarity 0.89)
- Major-Minor Mean Field Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.