Research map / Reinforcement learning / Policy gradient and actor-critic

Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Continuous Q-Score Matching (CQSM), a continuous-time, time-discretization-free Q-learning method that links diffusion policy scores to the action gradient of a martingale-characterized Q-function, and validates it on linear-quadratic control against policy gradient and little q-learning.

Problem

Q-learning and value-based RL are hard to extend to stochastic continuous-time control with continuous actions; existing Q-functions collapse to action-independent values and prior diffusion-policy score methods rely on time discretization.

Approach

Characterize continuous-time Q-functions via a martingale condition over an enlarged filtration (state and action noise), prove a score improvement theorem showing setting the policy score to lambda^{-1} times the action gradient of Q monotonically improves Q, and fit the score by regressing onto that action gradient, yielding an actor-critic algorithm (CQSM) without time discretization.

Key results

• On LQ control with step sizes 0.01, 0.1, and 1, CQSM achieves higher running average reward than policy gradient and little q-learning early in training, with all methods converging to similar levels later.
• Learned Q-function and score parameters approach the analytical optima, except theta_2, theta_3 and v_2, which the authors attribute to dual estimation of Q and its action gradient.

Measured effects reported in the paper

Surprising findings

• Dual estimation of the Q-function and its action gradient introduces extra variance and bias that impairs some parameter convergence.

Limitations

• Empirical evaluation limited to a one-dimensional linear-quadratic setting with analytical solutions; no high-dimensional or standard RL benchmarks.
• Admissibility and well-posedness conditions are assumed; convergence rate of CQSM not analyzed.
• Estimating both Q and its action gradient adds variance and bias, leading to inaccurate policy updates in some parameters.

Experimental setup

Datasets: simulated linear-quadratic control environment

Benchmarks: linear-quadratic stochastic control (simulated)

Metrics: running average reward parameter convergence to analytical optimum

Baselines: CT-RL policy gradient CT-RL little q-learning

Builds on: Q-learning little q-learning continuous-time policy gradient diffusion Q-learning DDPM

Closest papers by meaning

• Score as Action: Fine Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning (ICML 2025 · similarity 0.91)
• Learning a Diffusion Model Policy from Rewards via Q-Score Matching (ICML 2024 · similarity 0.90)
• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.90)
• Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.88)
• Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network (ICML 2025 · similarity 0.88)
• A Temporal Difference Method for Stochastic Continuous Dynamics (NeurIPS 2025 · similarity 0.88)