Research map / Reinforcement learning / Offline RL
Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization
Simon Sinong Zhan, Qingyuan Wu, Philip Wang, Frank Yang, Xiangyu Shi, Chao Huang, Qi Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train belief jointly with the policy instead of pretraining it separately (ablation) | DT-CORL · Hopper (delay 16) | normalized return: 68.3 -> 98.5 | Improved |
| train belief jointly with the policy instead of pretraining it separately (ablation) | DT-CORL · Hopper (delay 4) | normalized return: 92.1 -> 101.2 | Improved |
| use transformer belief with offline RL instead of raw state augmentation | CQL · Hopper-medium (deterministic delay 16) | normalized return: 21.1 -> 71.8 | Improved |
| use DT-CORL instead of Belief-CQL (aligned belief and value learning) | Belief-CQL · Walker2d-medium (deterministic delay 16) | normalized return: 39.2 -> 86.8 | Improved |
| use DT-CORL instead of Belief-CQL at short delay | Belief-CQL · HalfCheetah-medium (deterministic delay 4) | normalized return: 49.2 -> 47.4 | No clear effect |
| use DT-CORL instead of online delayed RL method DBPT-SAC offline | DBPT-SAC · AntMaze umaze | normalized return: 0.0 -> 76.7 | Improved |
| use belief-based policy with IQL instead of CQL | Belief-IQL vs Belief-CQL · Hopper-medium (deterministic delay 16) | normalized return: 42.9 -> 25.4 | Hurt |
| use DT-CORL on harder AntMaze tasks | DT-CORL · AntMaze medium-play and large-play | normalized return: 2.33 vs 5.67 | No clear effect |
| use stochastic instead of deterministic delay | DT-CORL · AntMaze umaze (delay 16) | normalized return: 40.0 -> 67.3 | Improved |
| use stochastic instead of deterministic delay | Augmented-CQL · AntMaze umaze (delay 8) | normalized return: 44.0 -> 37.0 | Hurt |
| increase transformer belief size (64x10 to 256x10) (ablation) | transformer belief · Hopper-medium-v2 (delay 16) | belief MSE: 2.68 -> 0.32 | Improved |
| increase delay to 64 steps | DT-CORL · Hopper-medium-v2 (deterministic delay 64) | normalized return: 2.77 vs 2.71 | No clear effect |
Closest papers by meaning
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.92)
- Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief (NeurIPS 2022 · similarity 0.91)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.91)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
