Research map / Multi-agent systems and games / Games and equilibria
Learning Strategy Representation for Imitation Learning in Multi-Agent Games
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STRIL, a plug-in framework that learns per-trajectory strategy representations with a P-VRNN (no player IDs), derives Randomness Indicator and Exploited Level scores, and filters offline multi-agent game data to improve BC, IQ-Learn and ILEED.
Problem
Offline imitation learning in multi-agent zero-sum games uses datasets mixing demonstrators of diverse skill and strategy, and standard IL treats all trajectories as homogeneous, so it learns sub-optimal behaviors; existing strategy-representation methods need player IDs or strong assumptions.
Approach
Learn a constant, trainable strategy representation per trajectory inside a partially-trainable conditional VRNN, then use the entropy of its action predictions (RI) and a neighborhood-based reward estimate (EL) as indicators to filter out sub-optimal trajectories before IL.
Key results
- On Two-player Pong, EL filtering raised BC worst score from -0.832 to -0.343 and the oracle "Best" filter reached -0.044 for BC.
- On Limit Texas Hold'em, RI filtering raised IQ-Learn worst score from -3.652 to 0.667 and BC from -1.255 to 0.532 (RI) and 0.662 (EL).
- On Connect Four, EL filtering raised BC worst score from -0.353 to 0.471; RI and EL did not improve ILEED on Connect Four.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Filter the training data with RI | BC · Two-player Pong | WS: -0.832\pm0.011 -> -0.613\pm0.052 | Helped |
| Filter the training data with EL | BC · Two-player Pong | WS: -0.832\pm0.011 -> -0.343\pm0.036 | Helped |
| Filter the training data with RI | IQ-Learn · Two-player Pong | WS: -0.804\pm0.044 -> -0.601\pm0.008 | Helped |
| Filter the training data with EL | IQ-Learn · Two-player Pong | WS: -0.804\pm0.044 -> -0.254\pm0.063 | Helped |
| Filter the training data with RI | ILEED · Two-player Pong | WS: -0.711\pm0.070 -> -0.607\pm0.058 | Helped |
| Filter the training data with EL | ILEED · Two-player Pong | WS: -0.711\pm0.070 -> -0.458\pm0.118 | Helped |
| Filter the training data with RI | BC · Limit Texas Hold'em | WS: -1.255\pm0.123 -> 0.532\pm0.052 | Helped |
| Filter the training data with EL | BC · Limit Texas Hold'em | WS: -1.255\pm0.123 -> 0.662\pm0.011 | Helped |
| Filter the training data with RI | IQ-Learn · Limit Texas Hold'em | WS: -3.652\pm0.428 -> 0.667\pm0.097 | Helped |
| Filter the training data with RI | ILEED · Connect Four | WS: 0.250\pm0.034 -> 0.267\pm0.081 | No clear effect |
| Filter the training data with EL | ILEED · Connect Four | WS: 0.250\pm0.034 -> 0.203\pm0.060 | Hurt |
Surprising findings
- On Connect Four, RI and EL filtering did not improve ILEED, which the authors attribute to the dataset already matching ILEED's assumptions.
- Filtering by randomness can be harmful when demonstrators use deterministic but poor strategies (noted as a limitation).
Limitations
- RI estimation fails when demonstrators use deterministic but poor strategies.
- EL estimation fails when reward-labeled trajectories are biased or cover a small region of the representation space.
Experimental setup
Datasets: Two-player Pong self-play dataset Limit Texas Hold'em dataset Connect Four self-play dataset
Benchmarks: Two-player Pong Limit Texas Hold'em Connect Four
Metrics: Worst Score over demonstrator set
Baselines: BC IQ-Learn ILEED Best (oracle dominant-demonstrator filtering)
Builds on: VRNN ILEED IQ-Learn Play-LMP
Closest papers by meaning
- Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.90)
- Efficient Active Imitation Learning with Random Network Distillation (ICLR 2025 · similarity 0.86)
- MisoDICE: Multi-Agent Imitation from Mixed-Quality Demonstrations (NeurIPS 2025 · similarity 0.86)
- Towards Building Human-like Smart Agents in Modern 3D Video Games (Student Abstract) (AAAI 2025 · similarity 0.86)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.85)
- Learning Equilibria from Data: Provably Efficient Multi-Agent Imitation Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.