Research map / Reinforcement learning / Policy gradient and actor-critic

Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean Field Games

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MF-TRPO (exact and sample-based variants) for computing approximate mean-field Nash equilibria in ergodic MFGs with finite state-action spaces, giving finite-sample convergence bounds and a grid-crowd experiment.

Problem

Computing approximate mean-field Nash equilibria (MFNE) in ergodic mean-field games with finite state-action spaces, with finite-sample guarantees under weaker assumptions than prior proximal-method analyses.

Approach

Extends TRPO (entropy-regularized, closed-form softmax mirror-descent update) to the mean-field game setting by alternating a TRPO best-response inner loop with an outer population-distribution update, using a discounted approximation of the ergodic problem and a nu-restart sampling oracle for a model-free variant.

Key results

• Sample-based MF-TRPO requires at most O~(1/epsilon^6) environment interactions to reach an epsilon-MFNE, per the paper's contribution statement.
• Exact MF-TRPO achieves an O~(1/L) rate inherited from TRPO, with exploitability bound eps_K = C exp(-tau/4 sum beta_j) + C sqrt(log L / L).
• In the 5x5 grid crowd experiment, exploitability of the sample-based algorithm converges after a few iterations (averaged over 10 and 3 seeds for eta=0.05 and 0.3).

Limitations

• Analysis restricted to finite state-action spaces; continuous spaces left to future work.
• Sample complexity of O~(1/epsilon^6) is high.
• Experiments are small-scale (5x5 grid) and mainly validate the sample-based variant.

Experimental setup

Datasets: 5x5 grid-based crowd modeling environment (synthetic)

Benchmarks: 5x5 Grid-Based Crowd Modeling

Metrics: exploitability

Builds on: TRPO Online Mirror Descent for MFG entropy-regularized RL mean-field MDP framework

Closest papers by meaning

• Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL (ICML 2024 · similarity 0.90)
• Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games (NeurIPS 2026 · similarity 0.90)
• Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation (ICLR 2025 · similarity 0.89)
• Bounded Rationality Equilibrium Learning in Mean Field Games (AAAI 2025 · similarity 0.89)
• Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics (NeurIPS 2025 · similarity 0.89)
• Learning Discrete-Time Major-Minor Mean Field Games (AAAI 2024 · similarity 0.88)