Research map / Reinforcement learning / Policy gradient and actor-critic

Revisiting a Design Choice in Gradient Temporal Difference Learning

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Revisits the discarded A^T TD idea from GTD's derivation, proposing A_t^T TD (single-sample estimate of A^T at a gap-delayed time with one parameter set and one learning rate), with asymptotic convergence proof and finite-sample analysis for a projected variant.

Problem

Off-policy policy evaluation with linear function approximation suffers from the deadly triad; GTD solves it but needs two parameter sets and two learning rates that are hard to tune, while the earlier A^T TD idea was discarded for O(K^2) cost.

Approach

Estimate A^T using a single sample taken at a future time t+f(t) with an increasing gap function f, which decorrelates the two samples and yields O(K) per-step cost with only one weight vector and one learning rate, at the price of O(f(t)) memory.

Key results

• A_t^T TD converges almost surely to the TD fixed point under a gap function f(t)=floor(ln^2(t+1)) and polynomially decaying learning rates (Theorem 1).
• Projected variant achieves E||w_t - w*||^2 = O(f(t) ln(t)/t), on par with on-policy linear TD up to logarithmic factors.
• On Boyan's chain and Baird's counterexample (10 seeds), the method matches GTD2, TDC and TDRC in final RMSPBE, and stays convergent off-policy; empirically it still converges with f(t)=O(log t), which violates the theoretical assumption.

Measured effects reported in the paper

Surprising findings

• The theoretical gap-function condition appears overly conservative, since logarithmic gap functions still converge empirically.

Limitations

• Memory grows with the gap function (Omega(ln^2 t)), which the paper argues is negligible but is unbounded.
• Empirical study is limited to two small benchmark tasks and described as preliminary.
• Finite-sample analysis is given only for a projected variant, not the original algorithm.

Experimental setup

Datasets: Boyan's chain Baird's counterexample

Benchmarks: Boyan's chain Baird's counterexample

Metrics: RMSPBE convergence rate

Baselines: off-policy TD GTD2 TDC Vtrace HTD TDRC

Builds on: GTD A^T TD TDC TD learning Preconditioned TD

Closest papers by meaning

• Temporal Difference Learning: Why It Can Be Fast and How It Will Be Faster (ICLR 2025 · similarity 0.90)
• The Pitfalls of Regularization in Off-Policy TD Learning (NeurIPS 2022 · similarity 0.90)
• Discerning Temporal Difference Learning (AAAI 2024 · similarity 0.89)
• A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning (NeurIPS 2025 · similarity 0.88)
• ContraDiff: Planning Towards High Return States via Contrastive Learning (ICLR 2025 · similarity 0.87)
• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.87)