Research map / Optimization / Nonconvex and smooth optimization
Learning in Stackelberg Mean Field Games: A Non-Asymptotic Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AC-SMFG, a single-loop actor-critic with two Markovian sample paths for learning in Stackelberg mean field games, and proves the first non-asymptotic O~(k^{-1/2}) convergence to a stationary point under a gradient alignment assumption.
Problem
Policy optimization in Stackelberg mean field games, a bi-level problem where a single leader anticipates the best response of an infinite homogeneous follower population; prior methods are nested-loop, rely on restrictive independence assumptions, and lack finite-time guarantees.
Approach
A single-loop actor-critic that updates leader policy, representative follower policy, and mean field (via stochastic approximation on a second sample path) simultaneously with TD critics, analyzed via a coupled Lyapunov function under a gradient alignment condition and a PL condition on the follower's softmax policy.
Key results
- Theorem gives min over t<k of E[squared leader gradient norm] at rate O~(k^{-1/2}) using two samples per iteration, improving on O~(k^{-2/5}) of a two-timescale bi-level method under lower-level strong convexity.
- Across 30 runs in Market Entrance, Shop Positioning and Equilibrium Pricing, AC-SMFG gives significantly faster leader convergence than OneByOne, CuiKoeppl and MARL (ADAGE) baselines.
- In Equilibrium Pricing with continuous states, AC-SMFG with Gaussian mean field closely tracks ADAGE and outperforms other SMFG baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AC-SMFG instead of the compared methods | Leader policy learning · Market Entrance | convergence speed | Helped |
| Use AC-SMFG instead of the compared methods | Leader policy learning · Market Entrance | leader reward | Helped |
| Use AC-SMFG instead of the compared methods | Leader policy learning · Shop Positioning | convergence speed | Helped |
| Use AC-SMFG instead of the compared methods | Follower policy learning · Shop Positioning | convergence speed | Hurt |
| Use AC-SMFG instead of the compared methods | Leader and follower policies · Shop Positioning | rewards | Helped |
| Use AC-SMFG instead of existing SMFG algorithms | Leader and follower policies · Equilibrium Pricing | rewards | Helped |
| Use AC-SMFG instead of other SMFG baselines | Leader policy · Continuous Equilibrium Pricing | leader reward | Helped |
Surprising findings
- Leader reward gap between ADAGE and AC-SMFG in Equilibrium Pricing is attributed to the Gaussian mean-field assumption possibly being inaccurate.
Limitations
- Analysis requires sufficiently large entropy regularization tau, so the unregularized equilibrium is not recovered.
- Convergence is to a stationary point, not a global optimum, since the composite objective lacks gradient domination.
- Assumes homogeneous followers and finite state/action spaces for analysis; MFG approximates N-agent games only asymptotically.
Experimental setup
Datasets: MFGLib Left-Right MFGLib Beach Bar Equilibrium Pricing environment
Benchmarks: Market Entrance (Left-Right) Shop Positioning (Beach Bar) Equilibrium Pricing
Metrics: leader reward follower reward convergence speed
Baselines: OneByOne CuiKoeppl ADAGE
Builds on: Stackelberg mean field games single-loop bi-level optimization entropy-regularized MFG learning ADAGE OneByOne fictitious play
Closest papers by meaning
- Transition-Informed Reinforcement Learning for Large-Scale Stackelberg Mean-Field Games (AAAI 2024 · similarity 0.92)
- Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithms (AAAI 2022 · similarity 0.92)
- Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation (ICLR 2025 · similarity 0.90)
- Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games (NeurIPS 2026 · similarity 0.89)
- Learning Discrete-Time Major-Minor Mean Field Games (AAAI 2024 · similarity 0.89)
- Bounded Rationality Equilibrium Learning in Mean Field Games (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.