Research map / Reinforcement learning / Offline RL
Online and Offline Reinforcement Learning by Planning with a Learned Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MCTS Reanalyse loss instead of CRR loss with MCTS action selection (ablation) | MuZero offline · RL Unplugged Atari (46 games) | median normalized score: 172.5 -> 265.3 | Helped |
| use MCTS action selection instead of policy sampling (Reanalyse loss) (ablation) | MuZero Unplugged · RL Unplugged Atari (46 games) | median normalized score: 203.2 -> 265.3 | Helped |
| use Reanalyse loss instead of CRR loss for policy head (ablation) | MuZero · RL Unplugged Atari (46 games) | median normalized score: 155.6 -> 203.2 | Helped |
| use CRR loss instead of supervised BC loss (ablation) | MuZero policy head · RL Unplugged Atari (46 games) | median normalized score: 54.0 -> 155.6 | Helped |
| increase model unroll from 1 to 5 steps (value action selection) (ablation) | MuZero BC loss · RL Unplugged Atari (46 games) | median normalized score: 105.0 -> 159.2 | Helped |
| vary unroll depth for policy-sampling action selection (ablation) | MuZero BC loss · RL Unplugged Atari (46 games) | median normalized score: 60.6 / 61.4 / 54.0 | No clear effect |
| inject trajectory action into MCTS root for continuous actions (ablation) | MuZero Unplugged · DM Control humanoid.run | mean score: 3.3 -> 633.4 | Helped |
| inject trajectory action into MCTS root (ablation) | MuZero Unplugged · DM Control walker.stand | mean score: 735.6 -> 759.8 | Helped |
| use 5-step TD value target instead of 0-step search value target (ablation) | MuZero Reanalyse 99.5% · Atari 57 games 20M frames | median human-normalized score: 115.3 -> 126.6 | Helped |
| increase Reanalyse fraction from 50% to 99.5% (fewer frames) (ablation) | MuZero · Atari 57 games | median human-normalized score: 1331.7% -> 126.6% | Mixed |
| use ResNet v2 with LayerNorm and Adam (ablation) | MuZero sticky · Atari 57 games 200M | median human-normalized score: 692.9 -> 1006.4 | Helped |
| use sticky actions (ablation) | MuZero · Atari 57 games 200M | median human-normalized score: 741.7 -> 692.9 | No clear effect |
Closest papers by meaning
- Efficient Online Reinforcement Learning with Offline Data (ICML 2023 · similarity 0.93)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning (NeurIPS 2020 · similarity 0.92)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Hybrid Reinforcement Learning from Offline Observation Alone (ICML 2024 · similarity 0.90)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.