Research map / Reinforcement learning / Policy gradient and actor-critic

Temporal Difference Learning: Why It Can Be Fast and How It Will Be Faster

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Theoretical analysis of temporal difference (TD) learning's convergence speed and a proposed faster variant, based only on the abstract's framing of TD as a divergent-but-thriving algorithm.

Problem

TD learning is known to be potentially unstable (divergent) in function approximation settings, yet it remains widely used in RL; the paper asks why it is nevertheless fast in practice and how it can be made faster.

Approach

Explains TD's practical speed despite its instability and proposes modifications that make it faster; specifics are not available in the abstract provided.

Surprising findings

• TD learning continues to thrive in RL despite having been proven divergent in some settings.

Closest papers by meaning

• Discerning Temporal Difference Learning (AAAI 2024 · similarity 0.93)
• Revisiting a Design Choice in Gradient Temporal Difference Learning (ICLR 2025 · similarity 0.90)
• An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks (ICML 2024 · similarity 0.90)
• Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features (NeurIPS 2026 · similarity 0.89)
• On Double Descent in Reinforcement Learning with LSTD and Random Features (ICLR 2024 · similarity 0.89)
• The surprising efficiency of temporal difference learning for rare event prediction (NeurIPS 2024 · similarity 0.89)