Research map / Multi-agent systems and games / Games and equilibria
Configurable Mirror Descent: Towards a Unification of Decision Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes generalized mirror descent (GMD) with multiple historical policies and arbitrary convex regularizers, plus a zero-order meta-controller (CMD) that tunes hyper-parameters per evaluation measure, and introduces GameBench of 15 games across all decision-making categories.
Problem
Existing decision-making algorithms are category-specific (single-agent, cooperative, competitive, mixed cooperative-competitive) and no single algorithm or comprehensive, academic-friendly benchmark covers all categories, solution concepts, and evaluation measures.
Approach
Generalize mirror descent to use multiple historical policies and any convex regularizer solved numerically via Newton's method on the KKT dual, then tune regularization weights with a zero-order meta-controller that uses only the sign of performance differences (direction-guided random search) conditioned on the chosen evaluation measure.
Key results
- CMD converges to approximate optimal or Nash equilibrium across all 15 GameBench games, and matches or beats baselines in NashConv, outperforming MMD-KL in MCCKuhn-A and MCCKuhn-B.
- In Leduc poker, fixed-hyperparameter GMD fails to decrease NashConv while CMD does, showing the meta-controller is needed.
- Vanilla CFR is typically inferior to CFR+ and CMD; CFR+ leads at short horizons but CMD catches up for longer horizons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CMD instead of MMD-KL | Mirror descent on GameBench · GameBench games | iterations to reach a similar OptGap or NashConv | Helped |
| Use fixed uniform hyper-parameters in GMD instead of MMD-KL | Mirror descent on GameBench · Most GameBench games | iterations to reach a similar OptGap or NashConv | Helped |
| Add the meta-controller to GMD | GMD · MCCKuhn-A and MCCKuhn-B | NashConv | Helped |
| Add the meta-controller to GMD | GMD · Leduc poker | NashConv | Helped |
| Use CMD instead of the baselines | Mirror descent on GameBench · Other GameBench games | NashConv | No clear effect |
| Use CFR instead of CFR+ or CMD | GameBench algorithms · GameBench games | learning performance | Hurt |
| Use CFR+ instead of CMD | GameBench algorithms · GameBench games | learning performance | Mixed |
| Use direction-guided random search (DRS) instead of other tested meta-controllers | CMD meta-controller · GameBench games | convergence rate or final performance | Helped |
| Use a Bregman divergence other than KL | CMD · Different decision-making categories | learning performance | Mixed |
| Configure the meta-controller objective for CCEGap or social welfare instead of NashConv | CMD · GameBench CCEGap and social welfare evaluations | learning performance | Helped |
Surprising findings
- The KL divergence, though widely adopted, is not the optimal choice across all decision-making categories according to the RQ4 experiments.
Limitations
- The meta-controller samples multiple candidates per update, adding computational cost.
- No theoretical convergence analysis of CMD; the numerical policy update lacks closed-form guarantees.
- Limited to Bregman divergences whose derivative functions are known in closed form; neural Bregman divergences left for future work.
Experimental setup
Datasets: GameBench
Benchmarks: GameBench
Metrics: OptGap NashConv CCEGap social welfare
Baselines: MMD-KL MMD-EU CFR CFR+ GMD with fixed hyper-parameters RS GLDS GLD DGLDS
Builds on: magnetic mirror descent counterfactual regret minimization random search zero-order optimization OpenSpiel
Closest papers by meaning
- The Update-Equivalence Framework for Decision-Time Planning (ICLR 2024 · similarity 0.86)
- Constrained Exploitability Descent: An Offline Reinforcement Learning Method for Finding Mixed-Strategy Nash Equilibrium (ICML 2025 · similarity 0.86)
- Local and Adaptive Mirror Descents in Extensive-Form Games (NeurIPS 2024 · similarity 0.86)
- Adaptively Perturbed Mirror Descent for Learning in Games (ICML 2024 · similarity 0.85)
- Joint Scoring Rules: Competition Between Agents Avoids Performative Prediction (AAAI 2025 · similarity 0.85)
- Operator World Models for Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.