Research map / Reinforcement learning / Offline RL
A Connection between One-Step RL and Critic Regularization in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add critic regularization with lambda=1 to classifier actor-critic (cross-entropy critic) (ablation) | classifier actor critic · tabular gridworld | policy match with one-step RL: R^2 >= 0.999 | Helped |
| Use one-step RL (behavioral TD target) instead of multi-step critic regularization | offline RL on D4RL · D4RL benchmark datasets | Q-value underestimation of actual returns | Helped |
| Use CQL critic regularization (lambda) with moderate coefficient | CQL · D4RL benchmark datasets | Q-value lower-bound property | Mixed |
| Use CQL critic regularization (implicit actor regularization) | CQL · D4RL benchmark datasets | MSE between policy actions and dataset actions | Helped |
| Use CQL with large regularization coefficient (lambda=10) vs one-step RL in gridworld | CQL · randomized 100 gridworld MDPs | fraction of states matching one-step RL argmax actions | Helped |
Closest papers by meaning
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.92)
- Demonstration-Regularized RL (ICLR 2024 · similarity 0.90)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
- Sharp Analysis for KL-Regularized Contextual Bandits and RLHF (NeurIPS 2025 · similarity 0.90)
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.