Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Heterogeneous Adversarial Play in Interactive Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HAP, an adversarial teacher-student curriculum learning framework in which a task-generating teacher and a student policy co-evolve via minimax-style updates, evaluated on Minigrid, CRAFT, and Crafter, plus supervised CurBench tasks and a human study.
Problem
Automated curriculum learning for multi-task RL typically uses static or unidirectional task selection; self-play frameworks assume symmetric agents, which does not fit asymmetric teacher-student settings.
Approach
Formalize teacher-student interaction as a zero-sum minimax game where a teacher (task selector, softmax over task logits conditioned on student return history) is rewarded for student failures while the student maximizes task return, with entropy regularization, cold-start warm-up, and probability lower bounds for stability.
Key results
- On Minigrid, HAP reaches a general score of 0.527 versus 0.493 for DreamerV3 (best baseline) and 0.747 for humans.
- On CRAFT, HAP reaches a general score of 0.562 versus 0.516 for DreamerV3 and 0.802 for humans.
- In a 30-participant Prolific human study, expert and HAP-generated tutorials gave similar learning rates and final performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adversarial curriculum learning instead of DreamerV3 | HAP on Minigrid · Minigrid General | success rate: 0.493 -> 0.527 | Helped |
| Use adversarial curriculum learning instead of DreamerV3 | HAP on CRAFT · CRAFT Hard | success rate: 0.27 -> 0.31 | Helped |
| Use adversarial curriculum learning instead of DreamerV3 | HAP on Crafter · Crafter General | success rate: 0.697 -> 0.723 | Helped |
| Add the last 100 student history entries to the teacher (ablation) | HAP · Minigrid General | success rate: 0.487 -> 0.527 | Helped |
| Add the last 100 student history entries to the teacher (ablation) | HAP · Minigrid Hard | success rate: 0.11 -> 0.20 | Helped |
| Use a 1,000-entry history instead of the last 100 entries (ablation) | HAP · Minigrid General | success rate: 0.527 -> 0.510 | Hurt |
| Add entropy regularization to the teacher objective (ablation) | HAP · Minigrid General | success rate: 0.467 -> 0.527 | Helped |
| Add entropy regularization to the teacher objective (ablation) | HAP · Minigrid Hard | success rate: 0.11 -> 0.20 | Helped |
| Use cold-start warm-up before teacher involvement (ablation) | HAP · Minigrid General | success rate: 0.523 -> 0.527 | No clear effect |
| Enforce lower bounds on task selection probabilities (ablation) | HAP · Minigrid | convergence | Helped |
| Use asynchronous instead of synchronous teacher updates (ablation) | HAP · Minigrid | student performance | No clear effect |
Surprising findings
- Ablation shows removing lower bounds causes failure to converge on middle and easy tasks in Minigrid.
- Crafter results show HAP only comparable to DreamerV3 and below PPO on general score (0.723 vs 0.693 is higher, but gains are modest), with performance gains described as modest.
Limitations
- Gains on Crafter are modest, and the method still lags human performance substantially on hard tasks.
- Pure adversarial setup can cause training collapse due to pathological teacher task selection, requiring entropy regularization and cold-start.
- Human study uses a modified Minigrid with altered visuals and rewards and only 30 participants.
Experimental setup
Datasets: Minigrid CRAFT Crafter CIFAR-100 RTE (GLUE) CurBench
Benchmarks: Minigrid CRAFT Crafter CIFAR-100 RTE
Metrics: task success rate general weighted average score accuracy
Baselines: DQN A2C PPO SAC TD3 DreamerV3 TSCL EXP3 auto-curriculum Ordered curriculum TTCL SPL LRE
Builds on: PPO A2C TSCL EXP3 auto-curriculum Curbench
Closest papers by meaning
- Skill Expansion and Composition in Parameter Space (ICLR 2025 · similarity 0.85)
- Mimicking the Maestro: Exploring the Efficacy of a Virtual AI Teacher in Fine Motor Skill Acquisition (AAAI 2024 · similarity 0.85)
- Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents (ICML 2026 · similarity 0.85)
- SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning (NeurIPS 2025 · similarity 0.85)
- Diffusion-based Curriculum Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.