Research map / Reinforcement learning / Policy gradient and actor-critic

Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Distributional value learning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows distributional RL agents are sensitive to decision frequency in continuous-time RL, defines distributional action gaps and a distributional superiority (rescaled by h^q), and builds DSUP(q) and DAU+DSUP(q) algorithms validated on high-frequency option trading.

Problem

Distributional RL agents in continuous-time MDPs at high decision frequency: action-conditioned return distributions collapse as the time between actions shrinks, so action gaps vanish and control degrades.

Approach

Defines W_p distributional action gaps and a distributional superiority (a minimal-variance coupled difference between action-conditioned and policy return distributions, via quantile functions), rescaled by h^q (q=1/2 preserves W_p action gaps and distortion-risk-measure rankings), and learns it with quantile TD learning.

Key results

• DSUP(1/2) outperforms QR-DQN, DAU, DSUP(1) and DAU+DSUP(1/2) at all but the two lowest frequencies in risk-neutral option trading, and is the most consistent performer.
• In risk-sensitive option trading at 35 Hz, DSUP(1/2) with CVaR action selection is conclusively the best over QR-DQN.
• Theory: W_p distributional action gaps of action-conditioned returns shrink at rate h^(1/2) (tight bounds), slower than action-value gaps which shrink at rate h.

Measured effects reported in the paper

Surprising findings

• Distributional action gaps of action-conditioned return distributions vanish at rate h^(1/2), which is slower than the h rate for action-value gaps, so naive distributional Q-learning fails in a different way than expected.
• DAU+DSUP(1/2), which preserves both action gaps and W_p action gaps, does not yield the strongest performance and is inconsistent.

Limitations

• Results shown on a simulated option-trading environment only.
• Assumes action-independent rewards, a limitation of continuous-time DRL in general.
• DAU+DSUP(1/2) mean estimates are unstable at high frequency.

Experimental setup

Datasets: option-trading simulation with Euler-Maruyama discretization of stock dynamics

Benchmarks: high-frequency option trading simulator

Metrics: return across decision frequencies

Baselines: QR-DQN DAU

Builds on: QR-DQN Deep Advantage Updating Advantage Learning

Closest papers by meaning

• Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning (ICML 2025 · similarity 0.88)
• More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning (ICML 2024 · similarity 0.87)
• Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model (NeurIPS 2024 · similarity 0.87)
• Near-Optimal Distributionally Robust Reinforcement Learning with General $L_p$ Norms (NeurIPS 2024 · similarity 0.87)
• Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.87)