Research map / Reinforcement learning / Offline RL
Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
Jiachen Li, Edwin Zhang, Ming Yin, Qinxun Bai, Yu-Xiang Wang, William Yang Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SGD-based Rev. KL Reg with closed-form CFPI operator I_MG (Gaussian Mixture behavior) (ablation) | one-step offline RL on D4RL Gym-MuJoCo · D4RL Gym-MuJoCo (9 tasks) | Total score: 725.8 vs 671.2 | Improved |
| Modeling behavior policy as Gaussian Mixture instead of Single Gaussian in CFPI operator (ablation) | CFPI operator I · D4RL Gym-MuJoCo (9 tasks) | Total score: 725.8 vs 688.9 | Improved |
| Closed-form CFPI operator vs Easy BCQ sampling operator under the same Gaussian Mixture behavior (ablation) | one-step policy improvement · D4RL Gym-MuJoCo (9 tasks) | Total score: 725.8 vs 670.6 | Improved |
| Rev. KL Reg (SGD policy improvement) across seeds/stopping points (ablation) | Rev. KL Reg operator · D4RL Gym-MuJoCo (9 tasks) | Total score variability | Hurt |
| Apply CFPI single-Gaussian operator on top of trained IQL policy | IQL policy on AntMaze · D4RL AntMaze (6 tasks) | Total score: 373.3 vs 361.7 (train) and 344.7 (1M) | Improved |
| Iterative CFPI (I_MG) vs one-step on AntMaze | offline RL on AntMaze · D4RL AntMaze (6 tasks) | Total score: 400.0 vs 378.0 (IQL) | Improved |
| Using one-step CFPI I_MG vs prior SOTA on Gym-MuJoCo | one-step offline RL · D4RL Gym-MuJoCo (9 tasks) | Total score: 757.0 vs 716.0 (OnestepRL) | Improved |
| Using I_MG on antmaze-umaze-diverse and large tasks vs baselines | iterative I_MG on AntMaze · D4RL AntMaze umaze-diverse | Score: 58.6 vs 84.0 (CQL) | Hurt |
| Replace SGD with CFPI on Hopper-M-E-like settings: mode selection not used; I_MG Hopper-M-E differs from its tuned HP (ablation) | I_MG on Hopper-M-E · D4RL Gym-MuJoCo Hopper-M-E | score: 73.0 (ablation) vs 104.2 (main table) | Mixed |
Closest papers by meaning
- Offline RL Without Off-Policy Evaluation (NeurIPS 2021 · similarity 0.92)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian (ICLR 2023 · similarity 0.91)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
- Relaxed Stationary Distribution Correction Estimation for Improved Offline Policy Optimization (AAAI 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
