Research map / Reinforcement learning / Offline RL
Value-Distributional Model-Based Reinforcement Learning
Carlos E. Luis, Alessandro G. Bottero, Julia Vinogradska, Felix Berkenkamp, Jan Peters
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the proposed value-distributional critic loss (EQR target quantiles from model samples) to QR-MBPO (ablation) | QR-MBPO · DMC cartpole-swingup-sparse | episode return: QR-MBPO-loss matches EQR-SAC; other QR-MBPO variants share similar lower performance | Improved |
| Increase buffer size and data collected under the ensemble (QR-MBPO-big) (ablation) | QR-MBPO · DMC selected tasks | episode return | No clear effect |
| Use model-consistent rollouts in QR-MBPO (QR-MBPO-consistent) (ablation) | QR-MBPO · DMC selected tasks | episode return | No clear effect |
| Increase next state-action samples for target quantiles from 1 to 25-100 (ablation) | EQR-SAC · DMC cartpole-swingup-sparse | sample efficiency | Improved |
| Sample size 1 with optimistic objective (ablation) | EQR-SAC (OFU) · DMC tasks | episode return | Hurt |
| Use number of quantiles m=1 instead of larger m (ablation) | EQR-SAC · DMC tasks | episode return | Hurt |
| Use rollout length k=1 instead of longer (ablation) | EQR-SAC · DMC tasks | episode return | Hurt |
| Use optimistic objective instead of mean objective (ablation) | EQR-SAC · Mountain Car | performance | Improved |
| Add more optimism in policy optimization (ablation) | EQR-SAC · DMC dense-reward tasks | episode return | No clear effect |
| Add more optimism in policy optimization (ablation) | EQR-SAC · DMC sparse-reward tasks | episode return | Mixed |
| Use distributional QR critic instead of standard critic | MBPO · DMC benchmark | episode return: MBPO 0.1 -> QR-MBPO (mean) 476.2 on cartpole-swingup-sparse | Improved |
| Use quantile value representation instead of critic ensemble | QU-SAC · DMC benchmark | episode return: QU-SAC 310.6 -> EQR-SAC (mean) 566.4 on cartpole-swingup-sparse | Improved |
Closest papers by meaning
- Epistemic Bellman Operators (AAAI 2025 · similarity 0.90)
- Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation (ICML 2025 · similarity 0.89)
- Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model (NeurIPS 2024 · similarity 0.88)
- Bayesian Exploration Networks (ICML 2024 · similarity 0.88)
- Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.87)
- Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
