Research map / Reinforcement learning / Policy gradient and actor-critic
Learning Diverse Risk Preferences in Population-Based Self-Play
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RPPO, a PPO variant using an expectile Bellman operator to learn policies with tunable risk preferences, and RPBT, a population-based self-play method where agents with diverse risk levels are tuned via PBT-style exploit/explore.
Problem
Self-play RL agents optimize only expected win rate against current or past copies, leading to limited strategy diversity, local optima, and poor generalization to different opponents.
Approach
Introduce an expectile Bellman operator that interpolates between worst-case and best-case Bellman operators via a risk level tau, extend it multi-step with GAE-like averaging, and use it inside PPO (RPPO); population members get different risk levels, auto-tuned by PBT, so diversity comes from risk preferences rather than population-level diversity objectives.
Key results
- RPBT beats SP and PP head-to-head (64% and 56% in Slimevolley; 67% and 63% in SumoAnts).
- RPBT vs PSRO win rate 53% in Slimevolley and 53% in SumoAnts, vs MEP 64% and 58%, with comparable performance to DvD in Slimevolley and better in SumoAnts.
- In a 4x4 grid-world toy example, tau in [0.2,0.4] gives risk-averse, [0.5,0.7] risk-neutral, and [0.8,0.9] risk-seeking paths.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RPBT instead of population-based self-play (PP) | self-play agents · Slimevolley | win rate: 56% | Helped |
| Use RPBT instead of self-play (SP) | self-play agents · Slimevolley | win rate: 64% | Helped |
| Use RPBT instead of population-based self-play (PP) | self-play agents · Sumoants | win rate: 63% | Helped |
| Use RPBT instead of self-play (SP) | self-play agents · Sumoants | win rate: 67% | Helped |
| Use RPBT instead of MEP | population-based self-play · Slimevolley | win rate: 64% | Helped |
| Use RPBT instead of TrajeDi | population-based self-play · Slimevolley | win rate: 59% | Helped |
| Use RPBT instead of DvD | population-based self-play · Slimevolley | win rate: 48% | No clear effect |
| Use RPBT instead of RR | population-based self-play · Slimevolley | win rate: 60% | Helped |
| Use RPBT instead of PSRO | population-based self-play · Slimevolley | win rate: 53% | Helped |
| Use RPBT instead of MEP | population-based self-play · Sumoants | win rate: 58% | Helped |
| Use RPBT instead of TrajeDi | population-based self-play · Sumoants | win rate: 63% | Helped |
| Use RPBT instead of DvD | population-based self-play · Sumoants | win rate: 54% | Helped |
Surprising findings
- Population-based diversity methods MEP and DvD showed no visible behavioral diversity in t-SNE analysis, while RPBT agents were distinct.
- The tau=0.1 agent was too conservative and did not always reach the flag in the toy example.
Limitations
- Multi-step sample-form operator introduces extra bias when tau differs from 1/2, and requires maintaining a table that increases compute and storage.
- Evaluated only on two competitive games (Slimevolley and SumoAnts) with PPO backbones.
Experimental setup
Datasets: Slimevolley SumoAnts
Benchmarks: Slimevolley SumoAnts
Metrics: pairwise win rate ELO score t-SNE state visualization distance to fence and hitting angle (behavior analysis)
Baselines: SP PP MEP TrajeDi DvD Reward Randomization PSRO
Builds on: PPO Generalized Advantage Estimation Population-based Training Fictitious self-play Diversity via Determinants (DvD)
Closest papers by meaning
- Provable Self-Play Algorithms for Competitive Reinforcement Learning (ICML 2020 · similarity 0.90)
- DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization (AAAI 2024 · similarity 0.90)
- Trajectory Diversity for Zero-Shot Coordination (ICML 2021 · similarity 0.89)
- GAE Falls Short in Imperfect-Information Self-Play Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.88)
- Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.