Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Opponent Modeling based on Subgoal Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OMG, which infers an opponent's future subgoal (a future state embedding) from history with a CVAE and conditions the agent's Q-network on it, with optimistic and conservative subgoal selectors for cooperative and general-sum games.
Problem
Opponent modeling in multi-agent environments with unseen, fixed opponent policies; existing methods predict only the opponent's next actions, which is shortsighted and hard to generalize.
Approach
Replace action prediction with inference of the opponent's future subgoal states via a variational inference model (CVAE) whose prior is a pre-trained VAE over states; subgoals are chosen within an H-step horizon by maximizing (optimistic) or minimizing (conservative) the value function, so subgoals are shared across opponent policies and generalize better.
Key results
- On SMAC 8m test opponents, OMG-optim outperforms IQL, LIAM and Naive OM across homologue and non-homologue opponent groups (e.g., 70.5% win rate vs 21.6% for QMIX on 7 non-homologue opponents in an appendix table).
- In Foraging, OMG converges to similar score as baselines but with shorter episodes, since it predicts the opponent's goal.
- Supervised-subgoal ablation (OMG-supervised) underperforms the CVAE-based OMG, indicating variational inference helps handle opponent policy uncertainty.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OMG-optim instead of QMIX | SMAC 8m · 7 non-homologue opponents | win rate: 21.6% -> 70.5% | Helped |
| Use OMG-optim instead of QMIX | SMAC 8m · 6 homologue opponents | win rate: 65.6% -> 86.8% | Helped |
| Use OMG-optim instead of QMIX | SMAC 8m · 7 homologue opponents | win rate: 61.0% -> 90.2% | Helped |
| Infer opponent subgoals rather than model opponent actions | OMG on Predator-Prey · Predator-Prey | score | Helped |
| Use OMG-conservative rather than OMG-optimistic | OMG on Foraging · Foraging | score | Mixed |
| Replace the CVAE inference model with supervised MLP training (ablation) | OMG on Foraging · Foraging | score | Hurt |
| Select subgoals randomly from the future horizon instead of using value-based selection (ablation) | OMG on Foraging · Foraging | score | Hurt |
| Select the opponent's next state as the subgoal (ablation) | OMG on Foraging · Foraging | score | Hurt |
| Select the opponent's third future state as the subgoal (ablation) | OMG on Foraging · Foraging | score | Hurt |
| Use the mixed subgoal input g instead of only the inferred or prior subgoal (ablation) | OMG on Foraging · Foraging | score | Helped |
| Use horizon H = 10 (ablation) | OMG on Foraging · Foraging | score | Hurt |
| Use horizon H = 1 (ablation) | OMG on Foraging · Foraging | score | Mixed |
Surprising findings
- Conservative subgoal selection (OMG-conservative) performs on par with IQL on SMAC, while optimistic selection generalizes better to unseen collaborators, contrary to the general-sum intuition.
Limitations
- Cannot handle open multi-agent systems where agents enter and leave during interaction.
- No formal theoretical results; the sample-complexity analysis in the appendix is only illustrative.
- Subgoal hit ratio remains at a moderate level (around 0.02 to 0.16), making predicted subgoals hard to verify.
Experimental setup
Datasets: Foraging (8x8 gridworld) Predator-Prey SMAC (8m, 3s_vs_5z, 2c_vs_64zg)
Benchmarks: Foraging Predator-Prey SMAC
Metrics: score episode length win rate subgoal hit ratio
Baselines: Naive OM LIAM D3QN PPO IQL
Builds on: DRON LIAM D3QN QMIX PPO
Closest papers by meaning
- An Open-Ended Learning Framework for Opponent Modeling (AAAI 2025 · similarity 0.91)
- Generalized Intention Modeling in Multi-Agent Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning (ICML 2024 · similarity 0.88)
- Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement (ICML 2025 · similarity 0.88)
- Opponent Modeling with In-context Search (NeurIPS 2024 · similarity 0.88)
- Modeling Boundedly Rational Agents with Latent Inference Budgets (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.