AI Paper Atlas

Research map / Reinforcement learning / Policy gradient and actor-critic

Rethinking Discount Regularization: New Interpretations, Unintended Consequences, and Solutions for Regularization in Reinforcement Learning

Sarah Rathnam, Sonali Parbhoo, Siddharth Swaroop, Weiwei Pan, Susan A. Murphy, Finale Doshi-Velez

JMLR 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
plan with a uniform Dirichlet prior of equal magnitude on the transition matrix instead of a lower planning discount factormodel-based certainty-equivalence RL · 10-State Random Chain, River Swim, GridWorld (uneven data)lossImproved
use state-action-specific epsilon (closed-form MSE minimizer) instead of one global regularization parametermodel-based uniform-prior regularization · 10-State Random Chain, River Swim, GridWorldlossImproved
use state-action-specific epsilon instead of a global epsilon or discount regularizationmodel-based regularization · Cancer simulatorlossImproved
estimate T from data (plug-in) instead of using the true T when computing epsilon-star (ablation)state-action-specific regularization · tabular environmentslossHurt
replace discount regularization or uniform prior with a hand-chosen left/right prior using expert knowledgemodel-based regularization · River SwimlossImproved
replace lower discount factor with weighted-average Q update of equal epsilonFQI model-free · tabular environment, model-freelossNo clear effect
replace lower discount factor with uniform prior on T of equal epsilon, equal data per state-actionmodel-based RL · 10-State Random ChainlossNo clear effect
add state-action-specific epsilon in model-free FQIFQI · River Swim, 10-State Random Chain, GridWorldlossHurt
use true Q to compute epsilon-star (no estimation error) in model-free method (ablation)model-free state-action-specific regularization · 10-State Random Chain and GridWorldlossImproved
add state-action-specific epsilon via kernel regression and FVI instead of fixed global epsilonfitted value iteration · Continuous River SwimlossImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy