Research map / Reinforcement learning / Visual and representation RL

DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization

AAAI 2024 · Reinforcement learning › Visual and representation RL · approach: Hierarchical skills and priors · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DGPO, an on-policy PPO-based method that learns multiple diverse high-reward strategies in one run with a shared latent-conditioned policy, using an information-theoretic pairwise-diversity intrinsic reward and alternating diversity/extrinsic constraints.

Problem

Standard RL seeks a single optimal strategy, whereas many tasks benefit from a diverse set of high-reward strategies; prior diversity methods (RSPO, RPG, DIAYN) need multiple networks/training stages or yield poor diversity or sample efficiency.

Approach

Define a stricter pairwise diversity score (minimum KL between state marginals of any two latent-conditioned policies), lower-bound it via a learned discriminator into an intrinsic reward, and solve two constrained problems (diversity-constrained then extrinsic-reward-constrained) by masking the reward terms in a PPO-style update with a shared latent-conditioned network.

Key results

• In MPE Spread (easy) and Spread (hard), DGPO recovers all optimal solutions (4 and 2) and converges over 1.7x faster than RSPO in Spread (easy) and 15x faster in Spread (hard).
• MAPPO recovers only a single solution in each MPE setting, and DIAYN finds only 2/4 solutions in Spread (easy).
• On Atari Boxing, DGPO learns attacks from different sides, while baselines tend to stay in one corner; the paper reports diversity and reward trade-offs without a single headline number.

Measured effects reported in the paper

Surprising findings

• Mutual-information-based diversity can be artificially high when a few policies remain similar, causing the optimizer to stop pushing diversity (motivating the stricter pairwise metric).

Limitations

• Choosing the diversity threshold delta and target reward R_target requires hyperparameter sweeps and heuristics.
• The theoretical derivation is only approximate: the two constrained stages are implemented as a concurrent approximation.
• Diversity metric relies on expert-designed behavior embeddings (agent positions, center of mass), which may not generalize.

Experimental setup

Datasets: MPE Spread (easy) MPE Spread (hard) Atari Pong Atari Boxing SMAC 2s_vs_1sc SMAC 3m

Benchmarks: MPE Spread (easy) MPE Spread (hard) Atari Pong Atari Boxing SMAC 2s_vs_1sc SMAC 3m

Metrics: diversity score M_Div extrinsic return win rate convergence steps

Baselines: MAPPO DIAYN SMERL RSPO

Builds on: PPO DIAYN SMERL RSPO MAPPO

Closest papers by meaning

• Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality (ICLR 2023 · similarity 0.91)
• Learning Diverse Risk Preferences in Population-Based Self-Play (AAAI 2024 · similarity 0.90)
• Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.89)
• Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.88)
• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.88)
• DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.88)