Research map / Reinforcement learning / Policy gradient and actor-critic

On the Divergence of Differential Temporal Difference Learning without Local Clocks

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Constructs a counterexample showing differential TD for average-reward policy evaluation converges with a local clock for any eta>0 but can diverge with a global clock for some eta, closing an open problem.

Problem

Whether differential TD (average-reward off-policy policy evaluation) converges with a global learning-rate clock for every eta>0, given it is known to converge with a local clock; in discounted RL the two clocks have matched convergence behavior.

Approach

Characterizes the stability region of the global-clock differential TD ODE as a rank-one perturbation of a singular M-matrix (D_mu(I-P_pi) + eta d_mu e^T), derives an exact stability region via Routh-Hurwitz, and builds an explicit MDP family where the region excludes an interval (alpha, 3alpha), showing divergence with a global clock.</numHERE>

Key results

• Exact stability region for the counterexample is eta in (0, alpha) union (3alpha, infinity), with alpha = 1/550 for m=23; instability on (alpha, 3alpha).
• At eta=2alpha in the m=23 MDP, the global-clock differential TD trajectory grows while the local-clock run contracts to 0.
• Lemma: for sufficiently small eta, A_eta is positive stable, so a small-eta global-clock convergence result holds.

Surprising findings

• Differential TD with a global clock can diverge for some eta>0 even though it converges with a local clock for all eta>0, unlike the discounted setting.
• Random sampling was reported to fail to find such counterexamples, so they were constructed manually.

Limitations

• Divergence is shown for a specific constructed instance and eta range, not almost-sure divergence of the stochastic algorithm.
• Proving almost sure divergence of differential TD with a global clock is left open.

Experimental setup

Benchmarks: hand-constructed counterexample MDP with m=23

Metrics: norm of value estimate v_t distance from v_t to span of all-one vector

Baselines: differential TD with local clock

Builds on: differential TD differential Q-learning ODE method for stochastic approximation Routh-Hurwitz criterion

Closest papers by meaning

• Revisiting a Design Choice in Gradient Temporal Difference Learning (ICLR 2025 · similarity 0.84)
• Discerning Temporal Difference Learning (AAAI 2024 · similarity 0.84)
• On Double Descent in Reinforcement Learning with LSTD and Random Features (ICLR 2024 · similarity 0.84)
• An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks (ICML 2024 · similarity 0.83)
• Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features (NeurIPS 2026 · similarity 0.83)
• The surprising efficiency of temporal difference learning for rare event prediction (NeurIPS 2024 · similarity 0.83)