Research map / Reinforcement learning / Policy gradient and actor-critic
Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning
Harm Van Seijen, Mehdi Fatemi, Arash Tavakoli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use logarithmic Q-learning instead of regular Q-learning | Tile-coded chain task with function approximation · Chain task | optimal-policy performance | Improved |
| Increase reward magnitudes by a factor of 100 (ablation) | Regular Q-learning · Chain task | optimal-policy performance | No clear effect |
| Add rewards that push action-values up by 100 (ablation) | Regular Q-learning · Chain task | optimal-policy performance | No clear effect |
| Use logarithmic mapping for positive rewards only (ablation) | Logarithmic Q-learning · Chain task | action-gap deviation | Improved |
| Use logarithmic mapping for negative rewards only (ablation) | Logarithmic Q-learning · Chain task | action-gap deviation | Mixed |
| Use logarithmic mapping for both positive and negative rewards (ablation) | Logarithmic Q-learning · Chain task | action-gap deviation | Mixed |
| Add a constant bias before applying the logarithm to handle negative rewards (ablation) | Logarithmic Q-learning · Chain task | action-gap deviation | Hurt |
| Set the regular-space averaging step size to 1, removing the separate regular-space averaging correction (ablation) | Logarithmic Q-learning · Positive-reward chain task | RMS error | Hurt |
| Use lower discount factors, from γ=0.84 to γ=0.99, with LogDQN (ablation) | LogDQN · Six Atari games used for tuning | human-normalized score | No clear effect |
| Use LogDQN rather than DQN | DQN · 55 Atari games | human-normalized score | Improved |
| Use Huber loss rather than squared-error loss (ablation) | LogDQN · Atari | performance | No clear effect |
Closest papers by meaning
- Taylor Expansion of Discount Factors (ICML 2021 · similarity 0.90)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.86)
- A Reduction Framework for Distributionally Robust Reinforcement Learning under Average Reward (ICML 2025 · similarity 0.86)
- Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.86)
- Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
