Research map / Reinforcement learning / Policy gradient and actor-critic

Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Sets

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes elliptic (multi-focal, sum-of-distances) uncertainty sets for robust RL to model directional market impact, derives implicit and explicit worst-case transition solutions, and uses robust TD learning to trade single assets and portfolios on LOBSTER-based simulated impact.

Problem

RL trading agents are trained on historical data without market impact, but deployment involves their own trades shifting prices; symmetric robust RL uncertainty sets (l_p balls) fail to capture the directional nature of this impact and can be overly conservative.

Approach

Generalize l_p-norm ball uncertainty sets to elliptic sets defined by a sum of norm distances to multiple foci, and derive implicit (via Fenchel duality and a dual variable) and explicit closed-form worst-case transition solutions for N=1 (general p), and N=2 with p=1 or p=2, enabling robust TD learning without inner optimization loops.

Key results

• On META, the l_p-ellipse robust RL achieved a Sharpe ratio of 2.48 versus 1.74 for non-robust RL, -0.28 for l_p-ball robust RL, and -0.95 for momentum.
• On MSFT, l_p-ellipse robust RL achieved Sharpe 1.20 (vs. momentum 1.10, non-robust RL -1.75, l_p-ball -0.82); on SPY it achieved 1.60 (vs. momentum 1.69, which is higher).
• In the multi-asset volume experiment, the robust agent stayed stable and profitable at about 200M volume while the non-robust agent's relative portfolio gap increased sharply.

Measured effects reported in the paper

Surprising findings

• The momentum baseline achieved a higher Sharpe ratio than the proposed method on SPY (1.69 vs 1.60).
• The l_p-ball robust RL baseline underperformed the non-robust RL on Sharpe for META (-0.28 vs 1.74), suggesting symmetric robustness can be overly conservative.

Limitations

• Theory restricted to finite state and action spaces; empirical results extend to continuous settings without theoretical guarantees.
• Elliptic sets do not guarantee the perturbed distributions are absolutely continuous w.r.t. the nominal kernel; the method may still be overly conservative.
• Explicit closed-form solutions only hold for N=1 (any p) or N=2 with p=1 or p=2; general cases require solving convex problems.

Experimental setup

Datasets: LOBSTER limit order book data minute-level historical price data for META, MSFT, SPY

Benchmarks: single-asset intra-day trading on META, MSFT, SPY (Jun-Dec 2022) with simulated market impact multi-asset portfolio rebalancing under increasing trade volume

Metrics: Sharpe ratio annualized return max drawdown final value relative portfolio gap

Baselines: Momentum Non-Robust RL Robust RL (l_p-ball)

Builds on: l_p-norm uncertainty sets R-contamination model robust TD learning Minkowski sum Fenchel conjugate

Closest papers by meaning

• Augmented Lagrangian Risk-constrained Reinforcement Learning for Portfolio Optimization (Student Abstract) (AAAI 2025 · similarity 0.86)
• Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.85)
• AlphaQCM: Alpha Discovery in Finance with Distributional Reinforcement Learning (ICML 2025 · similarity 0.85)
• A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.85)
• Decision Making under the Exponential Family: Distributionally Robust Optimisation with Bayesian Ambiguity Sets (ICML 2025 · similarity 0.85)