Research map / Reinforcement learning / Policy gradient and actor-critic

A Temporal Difference Method for Stochastic Continuous Dynamics

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes differential TD (dTD), a model-free TD policy-evaluation method that rewrites the fixed-policy HJB equation so sampled finite differences replace the drift and diffusion terms, and tests it with PPO on Brax MuJoCo-style tasks.

Problem

Hamilton-Jacobi-Bellman based RL for continuous ODE/SDE systems normally needs known drift and diffusion coefficients, so it cannot be applied model-free; standard TD discards the continuity structure when approximated with samples.

Approach

Rewrite the fixed-policy HJB equation so that drift and diffusion are expressed as limits of sampled finite differences of the state, yielding a model-free TD target (dTD) that keeps continuity information in the expectation argument; a beta-mixture with standard TD stabilizes learning.

Key results

• beta-dTD outperforms or matches standard TD across Hopper, HalfCheetah, Ant and Humanoid at noise levels coef 0.00, 0.01, 0.05, with the final performance of TD and beta-dTD tending to converge.
• beta-naive-dTD performs significantly worse than standard TD in all environments, even with tuned beta values (0.02 to 0.23).
• The fixed-policy HJB dynamics are shown to converge exponentially in L2 to the unique fixed point under bounded-domain, Neumann-boundary and uniform-ellipticity assumptions (Lax-Milgram argument).

Measured effects reported in the paper

Surprising findings

• The naive-dTD variant, which treats V(s_t) as the prediction, performs much worse than standard TD even though its tuned beta is small.
• Final performance of beta-dTD and TD tends to converge; the main benefit observed is faster improvement rather than a higher final return.

Limitations

• Convergence proof is for idealized continuous-time dynamics with a function V updated directly, not for function-approximated deep RL.
• Assumptions (bounded domain, Neumann boundary, uniform ellipticity) are hard to verify or maintain with deep RL function approximation.
• Method applies only to policy evaluation under on-policy sampling and is not applicable to off-policy or Q-learning settings.

Experimental setup

Datasets: Brax Hopper Brax HalfCheetah Brax Ant Brax Humanoid

Benchmarks: Brax Hopper Brax HalfCheetah Brax Ant Brax Humanoid

Metrics: episode return

Baselines: TD naive-dTD

Builds on: PPO A2C Bellman expectation equation Hamilton-Jacobi-Bellman equation

Closest papers by meaning

• Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
• Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning (ICML 2022 · similarity 0.89)
• Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control (NeurIPS 2025 · similarity 0.88)
• Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning (ICML 2025 · similarity 0.86)
• Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.86)
• Deep learning for continuous-time stochastic control with jumps (NeurIPS 2025 · similarity 0.86)