Research map / Reinforcement learning / Offline RL
Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoWorld, a model-based offline visual RL method that uses a paired online source simulator's world model and critic (aligned via state and reward alignment) to apply a min-max value constraint that mitigates overestimation on offline visual control tasks.
Problem
Offline visual reinforcement learning suffers from representation overfitting on limited high-dimensional image data and value overestimation on out-of-distribution actions, while existing conservative methods can over-constrain exploration.
Approach
Use an off-the-shelf online simulator from a related task as a 'test bed': train a separate source world model and critic, align its latent states with the target model (state alignment) and inject target reward info into the source reward model (reward alignment), then regularize the target critic with a min-max term taking the maximum of target and source value estimates, giving flexible rather
Key results
- On Meta-World, CoWorld (best-source) averages 4241 return across six transfer tasks versus 2730 for Offline DV2 and 3781 for DV2 Finetune.
- On DMC medium-replay datasets, CoWorld averages 488 versus 181 for Offline DV2 and 417 for DV2 Finetune, outperforming Offline DV2 by 169.6%.
- Ablations show performance drops when removing state alignment, reward alignment, or the min-max value constraint on RoboDesk Push Green Button.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoWorld with best-source selection instead of Offline DV2 | Meta-World cross-task offline RL · Meta-World cross-task (Avg.) | mean episode return: 2730 -> 4241 | Helped |
| Use CoWorld instead of DV2 Finetune | DMC offline RL · DMC offline (Avg.) | mean reward: 417 -> 488 | Helped |
| Use CoWorld instead of DV2 Finetune | DMC medium-expert offline RL · DMC medium-expert (Avg.) | mean reward: 738 -> 789 | Helped |
| Use low-dimensional source states with CoWorld transfer | RoboDesk Push Button and Open Slide · Meta-World → RoboDesk | mean return: 347 ± 24 -> 393 ± 64; 156 ± 46 -> 209 ± 43 | Helped |
| Use CoWorld with a sparse-reward source | RoboDesk transfer · Meta-World → RoboDesk | mean return: 314 ± 51 -> 335 ± 28; 173 ± 39 -> 184 ± 32 | Helped |
| Use separate domain-specific world models instead of one joint world model | CoWorld transfer to RoboDesk · Button Press → Push Button; Window Close → Open Slide | mean return: 342 ± 29 -> 428 ± 42; 173 ± 22 -> 202 ± 19 | Helped |
| Remove state alignment stage (ablation) | CoWorld · RoboDesk Push Green Button | performance | Hurt |
| Remove reward alignment stage (ablation) | CoWorld · RoboDesk Push Green Button | performance | Hurt |
| Remove min-max value constraint (ablation) | CoWorld · RoboDesk Push Green Button | performance | Hurt |
| Replace the min-max constraint with a direct value penalty (ablation) | CoWorld critic · Offline Meta-World | value estimation accuracy | Hurt |
| Use an excessively small domain KL loss scale β₂ (ablation) | CoWorld · Meta-World DC → BP | performance | Hurt |
| Use an excessively large domain KL loss scale β₂ (ablation) | CoWorld · Meta-World DC → BP | performance | Hurt |
Surprising findings
- Naively finetuning LOMPO from a source domain (792 average) performs worse than training it from scratch offline (1712), indicating negative transfer.
- Directly finetuning the source world model in the cross-environment Meta-World to RoboDesk setting does not yield significant gains.
Limitations
- Increased computational complexity from training an auxiliary source domain model and critic.
- Requires a related online simulator for the source domain, and gains depend on domain similarity.
- Validated mostly in simulation; real-robot experiments not conducted.
Experimental setup
Datasets: Meta-World RoboDesk DeepMind Control Suite D4RL-style medium-replay datasets
Benchmarks: Meta-World RoboDesk DeepMind Control Suite
Metrics: mean episode return standard deviation over seeds value estimation vs true discounted return
Baselines: Offline DV2 DrQ+BC CQL CURL LOMPO DV2 Finetune DV2 Finetune+EWC LOMPO Finetune
Builds on: DreamerV2 LOMPO CQL Elastic Weight Consolidation
Closest papers by meaning
- Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach (ICML 2025 · similarity 0.93)
- Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining (ICLR 2025 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.90)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.