Research map / Multi-agent systems and games / Games and equilibria
Advantage Alignment Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Advantage Alignment and Proximal Advantage Alignment, policy-gradient opponent-shaping algorithms that align agents' advantages, shown to subsume LOLA and LOQA, and evaluated on IPD, Coin Game, continuous Negotiation Game, and Melting Pot Commons Harvest Open.
Problem
Naive RL agents in general-sum social dilemmas converge to Pareto-suboptimal Nash equilibria; existing opponent shaping methods (LOLA, LOQA, POLA, COLA) are computationally heavy or hard to scale to continuous actions and complex environments.
Approach
Derive opponent shaping from two assumptions (agents maximize value; opponents act softmax over Q-values) to get a simple objective that increases an agent's action log-probability by the product of its discounted past advantages and the opponent's current advantage; show LOLA and LOQA implicitly perform this, and fold it into PPO as a modified advantage A*.
Key results
- On Commons Harvest Open, the best Advantage Alignment agent achieves on average 1.63 normalized per capita focal return, significantly outperforming all baselines.
- Advantage Alignment agents in the Coin Game cooperate with themselves and Always Cooperate while not being exploited by Always Defect, performing similarly to LOQA.
- In the one-step-history IPD, Advantage Alignment empirically yields a tit-for-tat-like policy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Advantage Alignment instead of the evaluated baselines | Commons Harvest Open agents trained with GTrXL · Commons Harvest Open | normalized focal return per capita: 1.63 | Helped |
| Use Advantage Alignment instead of PPO | Negotiation Game agents · Negotiation Game | cooperation and robustness to exploitation | Helped |
| Use Advantage Alignment instead of PPO | Coin Game agents · Coin Game | cooperation and robustness to exploitation | Helped |
| Replace the discount-based regularizer with a $1/(1+t)$ regularizer (ablation) | Advantage Alignment · Advantage Alignment implementation | performance | Hurt |
Limitations
- Relies on the assumption that opponents are softmax-rational over their Q-values.
- Theorem for LOLA equivalence requires orthonormal gradient basis assumption.
- Negotiation Game and IPD are modified or simplified versions, limiting direct comparability.
Experimental setup
Benchmarks: Iterated Prisoner's Dilemma Coin Game Negotiation Game (continuous variant) Melting Pot 2.0 Commons Harvest Open
Metrics: average return in league play normalized focal return per capita
Baselines: LOQA POLA MFOS PPO PPO with summed rewards acb vmpo opre
Builds on: LOLA LOQA POLA PPO DiCE
Closest papers by meaning
- Opponent Shaping in LLM Agents (ICLR 2026 · similarity 0.89)
- Opponent Modeling based on Subgoal Inference (NeurIPS 2024 · similarity 0.87)
- LOQA: Learning with Opponent Q-Learning Awareness (ICLR 2024 · similarity 0.86)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.86)
- An Open-Ended Learning Framework for Opponent Modeling (AAAI 2025 · similarity 0.86)
- Reciprocal Reward Influence Encourages Cooperation From Self-Interested Agents (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.