Research map / Reinforcement learning / Offline RL
Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, Xianyuan Zhan
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SQL instead of CQL for in-sample value learning | Offline RL · AntMaze medium and large small-data tasks | normalized return: 65.2 -> 75.1; 9.3 -> 24.2; 6.3 -> 36.7; 0 -> 34.2 | Improved |
| Use SQL instead of CQL for in-sample value learning | Offline RL · AntMaze medium and large small-data tasks | Bellman error: 13.1 -> 1.6; 64.4 -> 1.9; 30.6 -> 1.3; 300.5 -> 2.6 | Improved |
| Add SQL's sparsity term to value learning | SQL versus IQL · Noisy mixed expert/random MuJoCo datasets | normalized score | Improved |
| Vary SQL regularization parameter alpha (ablation) | SQL · MuJoCo mean | normalized score: 30.4 (alpha 0.5) -> 79.0 (alpha 10) | Mixed |
| Vary SQL regularization parameter alpha (ablation) | SQL · AntMaze mean | normalized score: 18.9 (alpha 0.2) -> 64.5 (alpha 0.5) -> 7.5 (alpha 2.0) | Mixed |
| Increase SQL alpha from 0.5 to 1 (ablation) | SQL · halfcheetah-medium | normalized score: 48.3 -> 48.7 | Improved |
| Increase SQL alpha from 0.5 to 2 (ablation) | SQL · walker2d-medium | normalized score: 22.3 -> 84.2 | Improved |
| Increase SQL alpha from 0.5 to 2 (ablation) | SQL · AntMaze medium-play | normalized score: 0.0 -> 80.2 | Improved |
| Use SQL instead of IQL | Offline RL · D4RL AntMaze and Kitchen benchmarks | normalized score | Improved |
| Use SQL instead of IQL | Offline RL · D4RL MuJoCo tasks | normalized score | No clear effect |
| Use SQL instead of IQL in Four Rooms | Offline RL with random behavior data · Four Rooms | policy success | Improved |
| Use SQL instead of CQL in small-data AntMaze | Offline RL · AntMaze small-data tasks | normalized return | Improved |
Closest papers by meaning
- Offline Reinforcement Learning with Implicit Q-Learning (ICLR 2022 · similarity 0.92)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.89)
- Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood (ICLR 2025 · similarity 0.89)
- Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
