Research map / Reinforcement learning / Offline RL
Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiveOff, an EM-style coordinate-ascent offline RL method that learns a latent-conditioned policy and posterior from a single task to discover multiple solutions, evaluated on new diverse D4RL locomotion datasets.
Problem
Discovering multiple distinct solutions (behaviors) from a single task in offline RL, where the dataset lacks latent skill labels.
Approach
Coordinate ascent resembling EM: an advantage-weighted E-step updates a latent-conditioned policy, and an M-step trains a posterior q(z|s,a) with advantage-weighted VAE-style objectives plus a mutual-information regularizer, jointly with the policy.
Key results
- DiveOff achieves the highest diversity score on several diverse-expert/diverse-medium-expert tasks (e.g., HopperVel diversity 0.88 vs 0.54 for best baseline on diverse-expert) while keeping high D4RL scores.
- On the hopper few-shot adaptation experiment, DiveOff obtains the best or near-best returns across four datasets (e.g., 1941.8 on LongHead for diverse-medium-replay).
- DiveOff (best latent) reaches 99.8 normalized score on Walker2dVel diverse-expert, comparable to single-solution baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiveOff instead of DiveOff + VAE, retaining advantage-weighted latent-representation learning (ablation) | DiveOff · diverse-expert and diverse-expert-medium locomotion tasks | diversity score | Helped |
| Add LSTM layers to the VAE posterior (ablation) | AWAC-L + VAE · diverse-medium and diverse-medium-replay locomotion tasks | normalized score | Hurt |
| Use the proposed mutual-information maximization strategy instead of the DIAYN auxiliary reward (ablation) | latent-conditioned offline RL · D4RL locomotion tasks with diverse behaviors | diversity score | Helped |
| Use the quantile-regression critic instead of double-clipped Q-learning (ablation) | DiveOff · D4RL locomotion tasks with diverse behaviors | normalized score | Mixed |
| Use TD3+BC policy updates with latent-conditioned behavioral cloning (ablation) | latent-conditioned offline RL · D4RL locomotion tasks with diverse behaviors | normalized score | Hurt |
| Select a high-return latent behavior instead of averaging behaviors | DiveOff · diverse locomotion tasks | normalized score | Helped |
Surprising findings
- IQL performs poorly on walker2d and hopper diverse-expert datasets despite strong expert-dataset performance in D4RL.
- Methods often failed to extract diverse behaviors from diverse-medium-replay datasets, attributed to lack of diverse successful states.
Limitations
- Learned behaviors vary across random seeds.
- Diversity of learned behaviors remains limited; the method does not cover all solutions.
- Added algorithmic and computational complexity relative to single-solution methods.
Experimental setup
Datasets: D4RL Walker2d D4RL Hopper D4RL HalfCheetah D4RL Ant (velocity-modified reward)
Benchmarks: D4RL Vel tasks (Walker2dVel, HopperVel, HalfcheetahVel, AntVel) few-shot adaptation on link-modified hopper
Metrics: normalized D4RL score determinantal diversity score with squared-exponential kerne few-shot return (K=25 budget) state entropy upper bound (GMM)
Baselines: CLUE AWAC-L+VAE IQL-L+VAE AWAC-L+VAE w/ LSTM AWAC-L+VAE+DIAYN BC TD3+BC IQL AWAC LAPO TD3+BC+VAE
Builds on: D4RL AWAC TD3+BC IQL DIAYN
Closest papers by meaning
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.90)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.90)
- Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach (ICML 2025 · similarity 0.89)
- Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.