Research map / Reinforcement learning / Offline RL
Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VeoRL, a model-based offline visual RL method that learns discrete latent behaviors from unlabeled videos and uses a two-stream (trunk and plan) world model with a latent-behavior-guided intrinsic reward to improve policy learning on Meta-World, CARLA, and MineDojo.
Problem
Offline visual RL struggles with suboptimal offline data, distribution shift from the target domain, and value overestimation; existing video-enhanced RL methods need action labels or same-domain videos.
Approach
Learn a discrete latent behavior codebook from unlabeled out-of-domain videos via vector quantization, then run a two-stream world model (trunk net with real actions, plan net with latent behaviors) and use the discrepancy between the two rollouts as an intrinsic reward and as conditioning for the actor and critic.
Key results
- On Meta-World Handle Press, VeoRL achieves 2650.9 episode return versus 1201.8 for DreamerV2 (success 0.60 vs 0.33).
- On CARLA, VeoRL outperforms DreamerV2 by as much as 350% in average episode return.
- On MineDojo Harvest water with bucket, success rises from 0.36 (DreamerV2) to 0.83 with VeoRL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VeoRL video-enhanced learning | DreamerV2 · Meta-World: Drawer Open | success rate: 0.18 → 0.70 | Helped |
| Add VeoRL video-enhanced learning | DreamerV2 · Meta-World: Handle Pull | success rate: 0.04 → 0.30 | Helped |
| Add VeoRL video-enhanced learning | DreamerV2 · Meta-World: Handle Press | success rate: 0.33 → 0.60 | Helped |
| Add VeoRL video-enhanced learning | DreamerV2 · MineDojo: Harvest log in plains | success rate: 0.10 → 0.35 | Helped |
| Add VeoRL video-enhanced learning | DreamerV2 · MineDojo: Harvest water with bucket | success rate: 0.36 → 0.83 | Helped |
| Add VeoRL video-enhanced learning | DreamerV2 · MineDojo: Harvest sand | success rate: 0.27 → 0.55 | Helped |
| Use fewer source-domain videos (ablation) | VeoRL · Meta-World: Handle Press | episode return: 2651 ± 620 → 2477 ± 441; 2651 ± 620 → 1859 ± 423 | Hurt |
| Train with random rather than medium-quality offline trajectories (ablation) | VeoRL · Meta-World: Button Press | episode return: 850 ± 169 → 638 ± 150 | Hurt |
| Add MMD loss for domain alignment (ablation) | VeoRL · Meta-World: Handle Press | success rate: 0.45 → 0.60 | Helped |
| Add MMD loss for domain alignment (ablation) | VeoRL · Meta-World: Handle Press | episode return: 1961 ± 585 → 2651 ± 620 | Helped |
| Remove latent behavior input from the policy network (ablation) | VeoRL · Meta-World: Handle Press, CARLA, and MineDojo: Harvest water with bucket | performance | Hurt |
| Remove latent behavior influence from the value network training objective (ablation) | VeoRL · Meta-World: Handle Press, CARLA, and MineDojo: Harvest water with bucket | performance | Hurt |
Surprising findings
- VIP and APV perform poorly on several tasks; APV yields an average CARLA return of about -500 and is excluded from the plot.
- DreamerV3 underperforms DreamerV2 in the offline setting without hyperparameter tuning.
Limitations
- Dual-branch world model increases computational complexity and memory requirements.
- Relies on the offline dataset being generated by a partially trained DreamerV2 agent; results on other data sources are limited.
- Some results rely on a few seeds (3 seeds) and small evaluation sets (50 episodes).
Experimental setup
Datasets: Meta-World BridgeData-V2 CARLA NuScenes MineDojo Minecraft YouTube videos
Benchmarks: Meta-World CARLA MineDojo
Metrics: success rate episode return
Baselines: DrQ+CQL LOMPO DreamerV2 APV VIP VPT DreamerV3
Builds on: DreamerV2 RSSM VQ-VAE LOMPO APV
Closest papers by meaning
- Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning (NeurIPS 2024 · similarity 0.93)
- Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining (ICLR 2025 · similarity 0.92)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.89)
- ODRL: A Benchmark for Off-Dynamics Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.