Research map / Reinforcement learning / Policy gradient and actor-critic
Single-Trajectory Distributionally Robust Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRQ, a fully model-free distributionally robust Q-learning algorithm that learns from a single trajectory under Cressie-Read f-divergence ambiguity sets, using a three-timescale stochastic approximation with asymptotic convergence guarantees.
Problem
Distributionally robust RL is hard to solve model-free from a single sample trajectory, since the robust Bellman operator is nonlinear and a one-sample plug-in estimator is biased; existing DRRL methods are model-based or need a simulator.
Approach
Dual reformulation of the Cressie-Read f-divergence robust Bellman operator yields a sup over a dual scalar eta; a three-timescale stochastic approximation updates auxiliary estimates Z1, Z2 (fastest), the dual eta (intermediate), and the Q table (slowest), removing the bias of one-sample estimators.
Key results
- On Cliffwalking, robust policies with rho=1.0 outperform non-robust Q-learning across perturbed wind probabilities p in {0.6,0.7,0.8}, while rho=1.5 is preferred at p=0.9 (100 seeds).
- On the American put option, DRQ clearly outperforms non-robust Q-learning when prices tend to rise (p=0.6, 0.7), and converges to the true DR value within about 10^5 samples, versus about 10^7 for Liu et al.
- Deep DDRQ achieves performance comparable to RFQI and SR-DQN under action and physical perturbations on CartPole and LunarLander, while DQN and FQI deteriorate quickly under slight perturbations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DRQ instead of non-robust Q-learning | Cliffwalking · Cliffwalking, perturbed environments | cumulative return | Helped |
| Increase the ambiguity-set radius to 1.0 (ablation) | DRQ · Cliffwalking, p in {0.6, 0.7, 0.8} | cumulative return | Helped |
| Increase the ambiguity-set radius to 1.5 (ablation) | DRQ · Cliffwalking, p=0.9 | cumulative return | Helped |
| Use an R-contamination ambiguity set instead of the Cressie-Read ambiguity set | Single-trajectory robust RL · Cliffwalking, p=0.8 and 0.9 | cumulative return | Hurt |
| Use DRQ instead of non-robust Q-learning | American put option · American put option, p={0.3, 0.4, 0.5} | average return | Hurt |
| Use DRQ instead of non-robust Q-learning | American put option · American put option, p={0.6, 0.7} | average return | Helped |
| Use DRQ instead of Liu's simulator-based DRQ | DRQ · American put option | samples to converge: 10^7 -> 10^5 | Helped |
| Use the model-based algorithm instead of DRQ | DRQ · American put option | samples to converge: less than 10^4 -> 10^5 | Hurt |
| Use DDRQ instead of non-robust DQN or FQI | CartPole and LunarLander · CartPole and LunarLander, perturbed environments | return | Helped |
| Use DDRQ instead of RFQI or SR-DQN | CartPole and LunarLander · CartPole and LunarLander, perturbed environments | return | No clear effect |
| Use DDRQ instead of RFQI | LunarLander · LunarLander, perturbed environments | return | Helped |
Surprising findings
- Robust policies outperform the non-robust policy even in the nominal Cliffwalking environment.
- The R-contamination robust baseline performs similarly to non-robust Q-learning and sometimes worse at high perturbation, due to over-conservatism.
Limitations
- Convergence proof is asymptotic and given for the synchronous case; asynchronous extension is only referenced.
- Deep DDRQ uses a two-timescale approximation that may bias the dual variable, so it may not be optimal for the primal problem.
- Relies on tabular or (s,a)-rectangular ambiguity sets and a choice of divergence and radius rho that must be tuned.
Experimental setup
Datasets: Cliffwalking American put option (simulated) CartPole LunarLander
Benchmarks: Cliffwalking American put option CartPole LunarLander
Metrics: average return steps to goal sample complexity (samples to converge) return under action and physical perturbations
Baselines: non-robust Q-learning R-contamination robust Q-learning Liu et al. DR Q-learning with simulator model-based DRRL algorithm DQN FQI RFQI soft-robust DQN (SR-DQN)
Builds on: Q-learning DQN two-timescale stochastic approximation multilevel Monte Carlo (Liu et al. DR Q-learning)
Closest papers by meaning
- Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.90)
- Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Time-Constrained Robust MDPs (NeurIPS 2024 · similarity 0.88)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Near-Optimal Distributionally Robust Reinforcement Learning with General $L_p$ Norms (NeurIPS 2024 · similarity 0.88)
- Model-Free Robust $\phi$-Divergence Reinforcement Learning Using Both Offline and Online Data (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.