Research map / Multi-agent systems and games / Cooperative multi-agent RL
A Generalist Hanabi Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R3D2, a distributed self-play RL agent that plays Hanabi through a text encoding of observations and actions, using a TinyBERT-plus-LSTM DRRN-style dueling Q-network, to generalize across player counts and partners.
Problem
Multi-agent RL agents for Hanabi overfit to one game setting (player count) and to their own training partners, failing at zero-shot coordination and transfer across settings.
Approach
Recast Hanabi observations and actions as text so the representation is setting-agnostic, and use a DRRN-style language-based network with a shared replay buffer so one agent can be trained across 2-to-5 player settings concurrently while learning robust self-play strategies.
Key results
- R3D2 achieves inter-XP scores substantially above R2D2, R2D2-OP and R2D2-OBL baselines in 2-player Hanabi while matching self-play and intra-XP performance.
- R3D2 intra-algorithm cross-play is on par with its self-play score, whereas R2D2 degrades sharply when paired with other seeds.
- GPT-4 reaches only 0-3 points in text-based Hanabi, and LoRA fine-tuned LLaMA-7B scores around 1 point.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on all player settings concurrently instead of a single setting | R3D2-M · Hanabi, 2- to 5-player settings | game score in cross-play | Helped |
| Use textual observations instead of vectorial observations (ablation) | R2D2-text · Hanabi, cross-play | game score | Helped |
| Use dynamic action-space processing with text-based state representation (ablation) | R3D2 · Hanabi, cross-play | game score | Helped |
| Initialize the language model with pretrained weights rather than random weights (ablation) | R3D2 · 2-player Hanabi | game score during training | Helped |
| Update the language model parameters during training (ablation) | R3D2 · 2-player Hanabi | game score during training | Helped |
| Reduce language-model update frequency from 1 to 2, 5, or 10 training steps per update (ablation) | R3D2 · 2-player Hanabi | game score during training | Hurt |
| Include discard-pile information in the observation (ablation) | BERT and DistilBERT Hanabi agents · Text-based Hanabi | game score | No clear effect |
| Increase the training dataset from 10% or less to 25% (ablation) | BERT Hanabi agent · Text-based Hanabi | game score | Helped |
| Increase the training dataset from 75% to 100% (ablation) | BERT Hanabi agent · Text-based Hanabi | game score | No clear effect |
| Increase model size from DistilBERT to BERT-base or BERT-large (ablation) | Text-based Hanabi language-model agent · Text-based Hanabi | game score | No clear effect |
| Use System Prompt 2 instead of System Prompt 1 (ablation) | GPT-4 · Text-based Hanabi | game score: 0.0 ± 0.0 -> 0.0 ± 0.0 | No clear effect |
| Use System Prompt 2 instead of System Prompt 1 (ablation) | o3-mini · Text-based Hanabi | game score: 7.66 ± 3.13 -> 12.0 ± 0.47 | Helped |
Surprising findings
- R3D2, trained purely by self-play, yields more robust cross-play than methods explicitly designed for zero-shot coordination.
- Including the discard pile in the text observation did not improve game score and doubled sequence length.
Limitations
- Cross-play evaluation uses R2D2-OP and R2D2-OBL checkpoints from original 2-player papers, so higher-player baselines are missing.
- Zero-shot coordination may lack realism compared with human partners.
- LLM prompting and fine-tuning baselines performed poorly, and the text pipeline inherits LM inference cost.
Experimental setup
Datasets: Hanabi self-play trajectories from OBL agent (44,000 state-action pairs)
Benchmarks: Hanabi (2-5 players)
Metrics: game score (max 25) self-play score intra-XP inter-XP zero-shot transfer score
Baselines: R2D2 (IQL) R2D2-OP R2D2-OBL R2D2-text GPT-4 o3-mini LLaMA-7B LoRA
Builds on: R2D2 DRRN Other-Play Off-Belief Learning DQN
Closest papers by meaning
- Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (ICLR 2020 · similarity 0.90)
- HyperMARL: Adaptive Hypernetworks for Multi-Agent RL (NeurIPS 2025 · similarity 0.88)
- Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents (ICML 2026 · similarity 0.88)
- FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.88)
- Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.87)
- Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Games (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.