Research map / Reinforcement learning / Offline RL
Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LOM, which models the offline behaviour policy as a Gaussian mixture (MDN), selects the highest-value mode via a behavioural hyper Q-function, and runs advantage-weighted imitation on that single mode, reporting gains on D4RL and multi-modal FetchReach/Push/Pick tasks.
Problem
Offline RL datasets often contain multi-modal action distributions from several behaviour policies; unimodal policies average conflicting actions into suboptimal or invalid actions.
Approach
Instead of modelling the full multi-modal behaviour distribution with expressive generative models, fit a GMM via an MDN, use a hyper Q-function to score each Gaussian mode by expected return, pick the best mode per state, and perform exponentiated-advantage weighted imitation only on samples from that mode.
Key results
- LOM achieves the highest mean in 12 of 15 D4RL tasks, e.g., 100.8 on hopper-medium vs 89.0 for the next best (WCVAE).
- On extremely multi-modal Fetch tasks, LOM averages 47.2 on FetchReach vs 42.3 for WCGAN and 32.4 for AWAC.
- The paper reports an average improvement of 53% over baselines on the Fetch multi-modal tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select and imitate the highest-reward Gaussian mode instead of imitating the full multi-modal behaviour distribution | LOM versus behaviour cloning, MDN, and AWAC · FetchReach, FetchPush, and FetchPickAndPlace | average score: 14.3_{\pm 3.4} -> 47.2_{\pm 3.1}; 11.8_{\pm 0.9} -> 44.3_{\pm 2.7}; 3.6_{\pm 0.8} -> 34.2_{\pm 3.6} | Helped |
| Vary the number of Gaussian mixture components M (ablation) | LOM · D4RL medium-replay | normalized score | Helped |
| Increase Gaussian mixture components beyond M = 10 (ablation) | LOM · D4RL full-replay | normalized score | No clear effect |
| Use too many Gaussian components, producing a very narrow selected mode (ablation) | LOM · FetchReach | action distribution coverage | Hurt |
| Use too few Gaussian components, producing broad modes (ablation) | LOM · FetchReach | action distribution fidelity | Hurt |
Surprising findings
- Removing the full-distribution modelling and imitating only a single selected mode outperforms modelling the entire multi-modal distribution with GAN/VAE weighted methods.
Limitations
- Number of mixture components M is a sensitive hyperparameter; too large yields overly narrow modes, too small yields OOD actions.
- Hyper Q-function Bellman target requires expectation over all hyper-actions, which may cause extrapolation errors.
- Identifying modes per state is computationally intensive when estimating M from data.
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d FetchReach multi-modal dataset FetchPush multi-modal dataset FetchPickAndPlace multi-modal dataset
Benchmarks: D4RL halfcheetah D4RL hopper D4RL walker2d FetchReach FetchPush FetchPickAndPlace
Metrics: normalised score average score
Baselines: OneStep AWAC CQL TD3BC LAPO DMPO WCGAN WCVAE
Builds on: TD3+BC advantage-weighted regression mixture density network performance difference lemma
Closest papers by meaning
- MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations (ICML 2023 · similarity 0.90)
- Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.88)
- Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data (AAAI 2025 · similarity 0.88)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.88)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.88)
- SEABO: A Simple Search-Based Method for Offline Imitation Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.