Research map / Multi-agent systems and games / Games and equilibria
LOQA: Learning with Opponent Q-Learning Awareness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LOQA, a decentralized opponent-shaping RL algorithm that assumes the opponent samples actions proportional to its Q-function, achieving reciprocity-based cooperation in IPD and Coin Game at lower compute than POLA and M-FOS.
Problem
Decentralized agents in general-sum games often converge to Pareto-suboptimal outcomes; existing opponent-shaping methods (LOLA/POLA, meta-learning) are computationally expensive due to differentiating through opponent learning steps or meta-game policies.
Approach
Assume the opponent's policy is a softmax over its action-value function Q2, estimate Q2 with Monte Carlo rollouts, and differentiate through this approximated opponent policy via a REINFORCE/loaded-DiCE term added to an actor-critic update, avoiding inner-loop opponent optimization and second-order gradients.
Key results
- In the 3x3 Coin Game, LOQA agents reach an average reward of 0.3 against themselves (vs 0.35 for always-cooperate pairs) and -0.05 against always-defect, comparable to POLA's -0.03.
- LOQA is fully trained in about 2 hours on one A100 versus 8 hours for POLA in the Coin Game comparison.
- In scalability tests LOQA reaches thresholds at least one order of magnitude faster in wall clock time than POLA and M-FOS, and all LOQA runs meet the strong threshold up to grid size 6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LOQA rather than POLA | Coin Game agent · Coin Game 3x3, against always defect | average reward: -0.05 vs -0.03 | No clear effect |
| Remove self-play during training (ablation) | LOQA · Coin Game 3x3, against itself | average reward | Hurt |
| Add self-play during training (ablation) | LOQA · Coin Game | cooperation with itself | Helped |
| Remove the replay buffer of past policy weights (ablation) | LOQA · Coin Game | performance | Hurt |
| Add a replay buffer of past policy weights (ablation) | LOQA · Coin Game | performance | Helped |
| Use LOQA rather than POLA | Coin Game agents · larger Coin Game grids | time to reach performance threshold: at least one order of magnitude | Helped |
Surprising findings
- Naive REINFORCE for the differentiable opponent Q-estimate gave suboptimal performance due to high variance, requiring loaded-DiCE.
- Trajectories tend to degenerate into homogeneous always-cooperate, reducing retaliation ability in the IPD.
Limitations
- Relies on the assumption that the opponent acts according to a softmax over an inner action-value function; cannot shape opponents violating this.
- Designed for discrete action spaces only.
- Normalization over action-values makes probabilities unable to saturate to 1 in IPD.
Experimental setup
Benchmarks: Iterated Prisoner's Dilemma Coin Game
Metrics: average reward (normalized return) wall clock time to reach thresholds probability of cooperation
Baselines: POLA M-FOS LOLA COLA SOS
Builds on: LOLA POLA Best Response Shaping DiCE loaded-DiCE
Closest papers by meaning
- Self-Play $Q$-Learners Can Provably Collude in the Iterated Prisoner's Dilemma (ICML 2025 · similarity 0.88)
- Multi-agent cooperation through learning-aware policy gradients (ICLR 2025 · similarity 0.88)
- Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.87)
- Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement (ICML 2025 · similarity 0.87)
- Maximizing utility in multi-agent environments by anticipating the behavior of other learners (NeurIPS 2024 · similarity 0.87)
- Advantage Alignment Algorithms (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.