Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Opponent Modeling with In-context Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OMIS, which pretrains an in-context Transformer (actor, opponent imitator, critic) for opponent modeling and then applies decision-time search with these components at test time to adapt to unknown non-stationary opponents.
Problem
Opponent modeling in multi-agent environments struggles to generalize to unknown opponent policies and suffers unstable adaptation when fine-tuning at test time.
Approach
Combine in-context learning pretraining of three conditioned components (best-response actor, opponent imitator, state-value critic) with gradient-free decision-time search that rolls out imagined opponent actions to choose actions, plus a confidence-based mixing rule between search and actor policies; theory links the pretrained model to posterior sampling and gives a policy improvement guarantee
Key results
- OMIS achieves higher average return and lower variance than baselines (Meta-PG, Meta-MAPG, MBOM, LIAM, MeLIBA, DRON, SP-MCTS) across [seen:unseen] ratios in Predator Prey, Level-Based Foraging, and OverCooked.
- SP-MCTS sometimes degrades OMIS w/o S (e.g., in OC and parts of PP), whereas OMIS's confidence-based mixing search improves it more consistently.
- A model-based variant (MBOMIS) using a learned transition model loses some performance relative to OMIS but still beats OMIS w/o S and most baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add decision-time search to the pretrained actor (ablation) | OMIS w/o S · Predator Prey, Level-Based Foraging, and OverCooked | average return | Helped |
| add confidence-based mixing of the search policy and original actor (ablation) | OMIS · Level-Based Foraging and OverCooked | average return | Helped |
| include episode-wise opponent context Depi (ablation) | OMIS w/o S · Predator Prey, Level-Based Foraging, and OverCooked | average return during pretraining | Helped |
| include step-wise opponent context Dstep (ablation) | OMIS w/o S · Predator Prey, Level-Based Foraging, and OverCooked | average return during pretraining | Helped |
| use learned transition dynamics instead of ground-truth dynamics | OMIS · Predator Prey, Level-Based Foraging, and OverCooked | average return | Hurt |
| use SP-MCTS rather than OMIS decision-time search | OMIS w/o S · OverCooked and some Predator Prey settings | average return | Hurt |
Surprising findings
- SP-MCTS search sometimes makes the base policy worse, unlike OMIS's search.
Limitations
- Only considers opponents switching among several fixed policies, not continuously learning or reasoning opponents.
- Focused on perfect-information games; imperfect information is not addressed.
- Uses a naive DTS method and assumes the ground-truth transition dynamics are available (learned dynamics reduce performance).
Experimental setup
Datasets: MEP opponent policy population trajectories on Predator Prey, Level-Based Foraging, OverCooked
Benchmarks: Predator Prey Level-Based Foraging OverCooked
Metrics: average return standard deviation across 5 seeds action prediction accuracy MSE of value estimates
Baselines: DRON LIAM MeLIBA Meta-PG Meta-MAPG MBOM SP-MCTS
Builds on: Posterior Sampling PPO MEP AlphaGo / MCTS Decision Transformer
Closest papers by meaning
- An Open-Ended Learning Framework for Opponent Modeling (AAAI 2025 · similarity 0.92)
- Opponent Modeling based on Subgoal Inference (NeurIPS 2024 · similarity 0.88)
- Offline Opponent Modeling with Truncated Q-driven Instant Policy Refinement (ICML 2025 · similarity 0.87)
- Generalized Intention Modeling in Multi-Agent Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Advantage Alignment Algorithms (ICLR 2025 · similarity 0.85)
- Agent-Omit: Adaptive Context Omission for Efficient LLM Agents (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.