Research map / Reinforcement learning / Visual and representation RL
Discovering Creative Behaviors through DUPLEX: Diverse Universal Features for Policy Exploration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DUPLEX, a context-conditioned diversity RL method that uses successor features with entropy regularization, dynamic intrinsic reward scaling and a soft lower bound to learn diverse near-optimal policies in GranTurismo 7 and multi-context MuJoCo, including OOD contexts.
Problem
Standard RL yields a single brittle solution; existing diversity methods (DOMiNO) and universal estimators struggle in highly dynamic, multi-context environments and out-of-distribution contexts.
Approach
Extends DOMiNO's repulsive diversity reward to context-conditioned policies, stabilizes intrinsic rewards with a dynamic scaling factor and a sigmoid soft lower bound on near-optimality, and estimates successor features with an SAC-style entropy term and averaged twin critics.
Key results
- In GranTurismo 7, only DUPLEX learns diverse policies within the near-optimal region, while DOMiNO collapses to similar behaviors.
- On single-task Walker2D, DUPLEX finds a better quality-diversity Pareto frontier than DOMiNOavg across optimality ratios.
- In OOD multi-context MuJoCo settings, DUPLEX outperforms UVFA and covers a wider range of diversity values while completing tasks; USFA and DOMiNO failed to learn in training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DUPLEX instead of DOMiNOavg | Diversity learning on Walker2D · Walker2D single-task | reward-diversity trade-off | Helped |
| Use DUPLEX instead of DOMiNOavg | Diversity learning in GranTurismo 7 · MountainTimeTrial | competitive policy diversity | Helped |
| Use DUPLEX instead of UVFA | Multi-context policy learning · Walker2D and Ant OOD contexts | reward | Helped |
| Add the soft lower bound (β ≠ 0) (ablation) | DUPLEX · GranTurismo 7 | competitive policy diversity | Helped |
| Set β = 0 instead of using the soft lower bound (ablation) | DUPLEX · GranTurismo 7 | lap completion | Hurt |
| Add entropy regularization to successor-feature estimation (ablation) | USFA · Multi-context MuJoCo | reward | Helped |
| Use the average instead of the minimum of the two critic estimates (ablation) | USFA · Multi-context MuJoCo | reward | Hurt |
| Use Lagrangian constrained optimization (ablation) | DUPLEX · GranTurismo 7 | competitive diverse behavior | Hurt |
Surprising findings
- Lagrangian constrained optimization was found ineffective in complex domains like GT, motivating the sigmoid soft lower bound instead.
- Adding entropy to the successor-feature objective alone significantly boosts vanilla USFA in multi-context MuJoCo.
Limitations
- Each extra policy adds cost from computing successor-feature distances across all policy pairs; scaling beyond about twenty policies is untested.
- GT experiments require about seven days per run and only three training runs, and the code and data are proprietary.
- DUPLEX does not impose an exploration strategy on each policy, so the degree of difference between policies is not directly controllable.
Experimental setup
Datasets: MuJoCo Walker2D MuJoCo Ant GranTurismo 7 MountainTimeTrial track
Benchmarks: GranTurismo 7 MountainTimeTrial MuJoCo Walker2D MuJoCo Ant multi-context MuJoCo Walker2D and Ant (OOD gravity and height contexts)
Metrics: extrinsic reward normalized diversity (Def. 4.1) lap times active policies
Baselines: DOMiNO USFA UVFA SAC QRSAC
Builds on: DOMiNO SAC USFA UVFA Successor features
Closest papers by meaning
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.87)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.86)
- Exploration by Learning Diverse Skills through Successor State Representations (NeurIPS 2024 · similarity 0.86)
- Beyond Single Stationary Policies: Meta-Task Players as Naturally Superior Collaborators (NeurIPS 2024 · similarity 0.86)
- Periodic Skill Discovery (NeurIPS 2025 · similarity 0.86)
- AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.