Research map / Reinforcement learning / Policy gradient and actor-critic
Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning
Kristopher De Asis, Alan Chan, Silviu Pitis, Richard Sutton, Daniel Graves
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one-step fixed-horizon TD instead of conventional TD | Baird's counterexample · Baird's counterexample | stability | Improved |
| Use a final horizon of 16 rather than 8 (ablation) | FHQ-learning · Slippery maze | mean episode length | Improved |
| Use a shorter horizon rather than the full episodic return (ablation) | FHQ-learning and Q-learning · Slippery maze | mean episode length | Improved |
| Use fixed horizons instead of conventional discount rates | FHQ-learning versus Q-learning · Slippery maze | mean episode length | Mixed |
| Use Deep FHQ-learning rather than DQN | LunarLander-v2 · LunarLander-v2 | return | Improved |
| Set the discount factor to 1 rather than 0.99 (ablation) | DQN · LunarLander-v2 | training stability | Hurt |
| Set the discount factor to 1 rather than 0.99 (ablation) | Deep FHQ-learning · LunarLander-v2 | performance sensitivity to discount factor | No clear effect |
| Add target networks (ablation) | Deep FHQ-learning · LunarLander-v2 | learning speed | Hurt |
| Add target networks (ablation) | DQN · LunarLander-v2 | area under the learning curve | Improved |
| Increase hidden-layer width beyond the reported setting (ablation) | Deep FHQ-learning · LunarLander-v2 | performance | No clear effect |
| Increase hidden-layer width beyond the reported setting (ablation) | DQN · LunarLander-v2 | performance | Hurt |
| Use a longer fixed horizon (ablation) | FHTD · 19-state random walk | root-mean-squared error | Mixed |
Closest papers by meaning
- Separating value functions across time-scales (ICML 2019 · similarity 0.90)
- Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.89)
- A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning (NeurIPS 2025 · similarity 0.88)
- Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.88)
- Discerning Temporal Difference Learning (AAAI 2024 · similarity 0.87)
- Revisiting a Design Choice in Gradient Temporal Difference Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
