AI Paper Atlas

Research map / Reinforcement learning / Offline RL

Offline Reinforcement Learning with Closed-Form Policy Improvement Operators

Jiachen Li, Edwin Zhang, Ming Yin, Qinxun Bai, Yu-Xiang Wang, William Yang Wang

ICML 2023 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace SGD-based Rev. KL Reg with closed-form CFPI operator I_MG (Gaussian Mixture behavior) (ablation)one-step offline RL on D4RL Gym-MuJoCo · D4RL Gym-MuJoCo (9 tasks)Total score: 725.8 vs 671.2Improved
Modeling behavior policy as Gaussian Mixture instead of Single Gaussian in CFPI operator (ablation)CFPI operator I · D4RL Gym-MuJoCo (9 tasks)Total score: 725.8 vs 688.9Improved
Closed-form CFPI operator vs Easy BCQ sampling operator under the same Gaussian Mixture behavior (ablation)one-step policy improvement · D4RL Gym-MuJoCo (9 tasks)Total score: 725.8 vs 670.6Improved
Rev. KL Reg (SGD policy improvement) across seeds/stopping points (ablation)Rev. KL Reg operator · D4RL Gym-MuJoCo (9 tasks)Total score variabilityHurt
Apply CFPI single-Gaussian operator on top of trained IQL policyIQL policy on AntMaze · D4RL AntMaze (6 tasks)Total score: 373.3 vs 361.7 (train) and 344.7 (1M)Improved
Iterative CFPI (I_MG) vs one-step on AntMazeoffline RL on AntMaze · D4RL AntMaze (6 tasks)Total score: 400.0 vs 378.0 (IQL)Improved
Using one-step CFPI I_MG vs prior SOTA on Gym-MuJoCoone-step offline RL · D4RL Gym-MuJoCo (9 tasks)Total score: 757.0 vs 716.0 (OnestepRL)Improved
Using I_MG on antmaze-umaze-diverse and large tasks vs baselinesiterative I_MG on AntMaze · D4RL AntMaze umaze-diverseScore: 58.6 vs 84.0 (CQL)Hurt
Replace SGD with CFPI on Hopper-M-E-like settings: mode selection not used; I_MG Hopper-M-E differs from its tuned HP (ablation)I_MG on Hopper-M-E · D4RL Gym-MuJoCo Hopper-M-Escore: 73.0 (ablation) vs 104.2 (main table)Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy