Research map / Reinforcement learning / Visual and representation RL

Explicit Exploration for High-Welfare Equilibria in Game-Theoretic Multiagent Reinforcement Learning

ICML 2025 · Reinforcement learning › Visual and representation RL · approach: Hierarchical skills and priors

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Ex2PSRO steers PSRO's strategy exploration toward high-welfare equilibria by behavior-cloning an exploration policy from high-welfare trajectories and regularizing best responses toward it via KL divergence, with no extra simulation cost.

Problem

Equilibrium selection in games with multiple equilibria: vanilla PSRO tends to converge to Pareto-dominated equilibria, so the goal is to find higher-welfare equilibria in multiagent RL.

Approach

Build a buffer of top-welfare trajectories from existing simulation data, behavior-clone an exploration policy from it, and regularize the SAC best-response toward that exploration policy via a KL term, so exploration is biased toward high-welfare strategies without extra simulations.

Key results

• On Harvest-Dense, Ex2PSRO reached welfare 150.2 ± 7.9 versus 119.6 ± 3.4 for vanilla PSRO (p < .01).
• On Bargaining γ=.99, Ex2PSRO reached 12.86 ± 0.26 versus 12.50 ± 0.55 for vanilla PSRO (p = .04); on γ=.95 it reached 14.02 ± 0.22 versus 13.76 ± 0.34 (p = .03).
• Combining GRO with Ex2PSRO raised welfare over GRO alone in Bargaining γ=.99 (13.49 ± .39 vs 12.99 ± .33, p < .01) and γ=.95 (15.24 ± .08 vs 15.07 ± .10, p < .01).

Measured effects reported in the paper

Surprising findings

• In the Harvest-Sparse benchmark, the MaxWelfare ablation performed similarly to Ex2PSRO (p = .32 for MaxWelfare vs Ex2PSRO comparison is reported as similar), and in Bargaining γ=.95 Uniform and PrevBR found lower-regret profiles than Ex2PSRO.
• Ex2PSRO's gains in Harvest are attributed to any regularization that reduces immediate apple collection, so the specific exploration policy matters less there than in Bargaining.

Limitations

• Hyperparameter tuning for Ex2PSRO was conservative compared with exhaustive tuning for vanilla PSRO; P-values are not adjusted for multiple hypotheses.
• Method assumes symmetric games; non-symmetric extension requires per-player exploration policies and payoff normalization.
• Welfare is measured relative to the game's equilibrium distribution only approximately, since computing it is intractable for large games.

Experimental setup

Datasets: Harvest (Gathering) simulation trajectories Bargaining game simulation trajectories

Benchmarks: Harvest-Dense Harvest-Sparse Bargaining gamma=.99 Bargaining gamma=.95

Metrics: equilibrium welfare regret welfare convergence over PSRO iterations Welch's t-test p-values

Baselines: Vanilla PSRO GRO Uniform exploration policy MinWelfare MaxWelfare PrevBR

Builds on: Policy Space Response Oracles Soft Actor-Critic Generalized Response Objectives Regularized replicator dynamics OpenSpiel

Closest papers by meaning

• Iterative Empirical Game Solving via Single Policy Best Response (ICLR 2021 · similarity 0.91)
• Global Policy-Space Response Oracles for Two-Player Zero-Sum Games (ICML 2026 · similarity 0.89)
• Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.87)
• Incentivize without Bonus: Provably Efficient Model-based Online Multi-agent RL for Markov Games (ICML 2025 · similarity 0.87)
• Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.87)
• A-PSRO: A Unified Strategy Learning Method with Advantage Metric for Normal-form Games (ICML 2025 · similarity 0.86)