Research map / Reinforcement learning / Offline RL
Conservative Q-Learning for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the conservative Q-value regularizer (CQL(H)) | Offline RL on AntMaze · antmaze-medium-play and antmaze-medium-diverse | normalized score: 0.0 -> 61.2; 8.0 -> 53.7 | Helped |
| Add the conservative Q-value regularizer (CQL(H)) | Offline RL on AntMaze · antmaze-large-play and antmaze-large-diverse | normalized score: 0.0 -> 15.8; 0.0 -> 14.9 | Helped |
| Add CQL(H) to offline Q-learning | Atari with 1% of DQN-replay data · Q*bert and Breakout | return: 383.6 -> 14012.0; 11.0 -> 61.1 | Helped |
| Add CQL(H) to offline Q-learning | Atari with 1% of DQN-replay data · Seaquest and Asterix | return: 672.9 -> 779.4; 386.5 -> 592.4 | Helped |
| Add CQL(H) to offline actor-critic | Adroit human-demonstration tasks · pen-human, hammer-human, door-human, relocate-human | normalized score: 34.4 -> 37.5; 1.5 -> 4.4; 0.5 -> 9.9; 0.0 -> 0.20 | Helped |
| Use previous policy as rho instead of the CQL(H) variant (ablation) | CQL on high-dimensional Adroit tasks · pen-human and relocate-human | normalized score: 37.5 -> 55.8; 0.20 -> 0.35 | Helped |
| Use previous policy as rho instead of the CQL(H) variant (ablation) | CQL on AntMaze · antmaze-medium-play and antmaze-medium-diverse | normalized score: 61.2 -> 4.6; 53.7 -> 5.1 | Hurt |
| Use previous policy as rho instead of the CQL(H) variant (ablation) | CQL on high-dimensional action tasks · Adroit tasks | training stability | Helped |
| Use the pointwise lower-bound objective instead of CQL(H) (ablation) | CQL policy-value estimation · hopper-medium-expert | predicted return minus true return: -43.20 -> -151.36 | Hurt |
| Use the pointwise lower-bound objective instead of CQL(H) (ablation) | CQL policy-value estimation · hopper-mixed and hopper-medium | predicted return minus true return: -10.93 -> -22.87; -7.48 -> -156.70 | Hurt |
| Use CQL(H) rather than unconstrained SAC | Offline control with mixed-policy data · D4RL mixed datasets | normalized return | Helped |
| Regularize Q-values rather than relying only on a policy constraint | Offline RL with narrow data distributions · hopper-expert | OOD-action Q-value gap | Helped |
Closest papers by meaning
- Mildly Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2022 · similarity 0.94)
- On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.92)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model (NeurIPS 2024 · similarity 0.90)
- Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.