Research map / Reinforcement learning / Offline RL
SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SeMOPO, a model-based offline RL method that separates endogenous and exogenous latent dynamics, uses conservative (trajectory-by-trajectory) sampling to train the model, and penalizes rewards with endogenous ensemble disagreement; also introduces the LQV-D4RL benchmark of low-quality visual datasets with moving distractors.
Problem
Offline model-based RL from high-dimensional visual data with low-quality (random or sub-optimal) behavior data and moving distractors, where model uncertainty estimates are biased by task-irrelevant distractor dynamics, leading to overly conservative policies.
Approach
Decompose the latent state into endogenous and exogenous parts under an EX-BMDP assumption, train the separated model with conservative sampling (one trajectory at a time, favoring deterministic behavior policies), and estimate model uncertainty and apply the reward penalty only on endogenous states.
Key results
- SeMOPO achieves the best normalized return on most LQV-D4RL tasks, e.g., Hopper Hop medium 1.24 ± 0.16 vs 0.58 for the next-best baseline shown (InfoGating), and Walker Walk random 0.77 vs 0.27 for Offline DV2.
- SeMOPO yields lower model uncertainty than Offline DV2 across all nine datasets, and conservative sampling yields significantly lower action entropy than random sampling.
- In the online-distraction generalization test, SeMOPO reaches 0.78 on Walker Walk random versus 0.33 for Offline DV2 and 0.05 for DrQ+BC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SeMOPO instead of the strongest listed baseline | Offline visual RL · LQV-D4RL (15 task-dataset settings) | normalized test return | Mixed |
| Use SeMOPO instead of the strongest listed baseline | Offline visual RL · Walker Walk and Cheetah Run with different online distractions | normalized test return | Helped |
| Use conservative sampling (Se+CS) instead of random sampling (Se+RS) (ablation) | Separated model training · LQV-D4RL (nine tasks) | comparative performance probability: >0.8 | Helped |
| Add dissociated reward prediction loss to separated-model training with random sampling (ablation) | Se+RS · LQV-D4RL (nine tasks) | comparative performance probability: ≈ 0.4 | No clear effect |
| Use separated model training with random sampling instead of Offline DV2 with random sampling (ablation) | Offline visual RL · LQV-D4RL (nine tasks) | comparative performance | Helped |
| Use conservative sampling instead of random sampling (ablation) | Separated model training · LQV-D4RL (nine datasets) | action entropy | Helped |
| Estimate uncertainty on endogenous states rather than the full latent state | SeMOPO vs Offline DV2 · LQV-D4RL (nine datasets) | model uncertainty | Helped |
| Add dissociated reward prediction to random sampling (ablation) | Separated model training · Cheetah Run medium_replay | task-relevant information in endogenous-state reconstruction | Mixed |
Surprising findings
- All methods, including SeMOPO, fail on Humanoid Walk (normalized returns near 0).
- Dissociated reward prediction on top of random sampling does not improve offline policy learning over random sampling alone.
Limitations
- Assumes independence between endogenous and exogenous transitions under EX-BMDP, which may not hold in real-world tasks.
- Fails on the Humanoid Walk task, with all methods near zero return.
- Conservative sampling is a heuristic; the paper notes the theory does not guarantee maximum likelihood via this sampling.
Experimental setup
Datasets: LQV-D4RL Kinetics (driving car videos as backgrounds) DMControl Suite Gym Car Racing V-D4RL
Benchmarks: LQV-D4RL V-D4RL
Metrics: normalized test return probability of superiority with 95% bootstrap confidence int model uncertainty action entropy
Baselines: Offline DV2 LOMPO DrQ+BC DrQ+CQL BC InfoGating
Builds on: MOPO DreamerV2 TIA LOMPO Block MDP
Closest papers by meaning
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · similarity 0.88)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Generative OOD-regularized Model-based Policy Optimization (NeurIPS 2026 · similarity 0.87)
- Neural Network Approximation for Pessimistic Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.