Research map / Reinforcement learning / Policy gradient and actor-critic
Rethinking Discount Regularization: New Interpretations, Unintended Consequences, and Solutions for Regularization in Reinforcement Learning
Sarah Rathnam, Sonali Parbhoo, Siddharth Swaroop, Weiwei Pan, Susan A. Murphy, Finale Doshi-Velez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| plan with a uniform Dirichlet prior of equal magnitude on the transition matrix instead of a lower planning discount factor | model-based certainty-equivalence RL · 10-State Random Chain, River Swim, GridWorld (uneven data) | loss | Improved |
| use state-action-specific epsilon (closed-form MSE minimizer) instead of one global regularization parameter | model-based uniform-prior regularization · 10-State Random Chain, River Swim, GridWorld | loss | Improved |
| use state-action-specific epsilon instead of a global epsilon or discount regularization | model-based regularization · Cancer simulator | loss | Improved |
| estimate T from data (plug-in) instead of using the true T when computing epsilon-star (ablation) | state-action-specific regularization · tabular environments | loss | Hurt |
| replace discount regularization or uniform prior with a hand-chosen left/right prior using expert knowledge | model-based regularization · River Swim | loss | Improved |
| replace lower discount factor with weighted-average Q update of equal epsilon | FQI model-free · tabular environment, model-free | loss | No clear effect |
| replace lower discount factor with uniform prior on T of equal epsilon, equal data per state-action | model-based RL · 10-State Random Chain | loss | No clear effect |
| add state-action-specific epsilon in model-free FQI | FQI · River Swim, 10-State Random Chain, GridWorld | loss | Hurt |
| use true Q to compute epsilon-star (no estimation error) in model-free method (ablation) | model-free state-action-specific regularization · 10-State Random Chain and GridWorld | loss | Improved |
| add state-action-specific epsilon via kernel regression and FVI instead of fixed global epsilon | fitted value iteration · Continuous River Swim | loss | Improved |
Closest papers by meaning
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.90)
- Demonstration-Regularized RL (ICLR 2024 · similarity 0.89)
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.89)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.88)
- Decoupling regularization from the action space (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
