Research map / Reinforcement learning / MDPs and planning
Time-Constrained Robust MDPs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces time-constrained robust MDPs (TC-RMDPs), where a parametric adversary's parameter can change only by a bounded amount per step, and derives three TC variants (Oracle-TC, Stacked-TC, vanilla TC) applied to RARL and M2TD3 for robust continuous control on MuJoCo.
Problem
Standard robust RL relies on rectangularity assumptions that let the adversary pick transition models independently across time steps and state-action pairs, producing overly conservative policies when disturbances are correlated over time.
Approach
Model the adversary as choosing a parameter psi whose per-step change is bounded by L (Lipschitz in time), so the adversary's action set is a ball B of radius L and the state is augmented with psi (oracle), previous state-action (stacked), or nothing (vanilla); this is applied as a modification of RARL and M2TD3 actor-critic training.
Key results
- Under worst-case time-constrained perturbation, TC-M2TD3 scores 2.9 aggregate normalized vs. 1.0 for M2TD3 and -0.08 for RARL (Table 1).
- In static worst-case evaluation, Oracle TC-M2TD3 averages 3.12 and Stacked TC-M2TD3 2.23 in the static average-case table, versus 1.0 for M2TD3.
- The paper reports that TC-trained policies achieve up to a 2.9-fold improvement over baselines under time-constrained perturbations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train M2TD3 with time-constrained perturbations and state-only policy input | M2TD3 · MuJoCo time-coupled worst case (Agg) | normalized performance: 1.0 ± 0.58 -> 2.9 ± 0.4 | Helped |
| Train RARL with time-constrained perturbations and state-only policy input | RARL · MuJoCo time-coupled worst case (Agg) | normalized performance: -0.08 ± 0.37 -> 1.46 ± 0.6 | Helped |
| Train M2TD3 with time-constrained perturbations and state-only policy input | M2TD3 · MuJoCo static worst case (Agg) | normalized performance: 1.0 ± 1.01 -> 1.02 ± 0.68 | Mixed |
| Train M2TD3 with time-constrained perturbations and state-only policy input | M2TD3 · MuJoCo static average case (Agg) | normalized performance: 1.0 ± 0.59 -> 2.91 ± 0.73 | Helped |
| Train M2TD3 with time-constrained perturbations and policy access to ψ | M2TD3 · MuJoCo time-coupled worst case (Agg) | normalized performance: 1.39 ± 0.31 -> 3.07 ± 0.81 | Helped |
| Train RARL with time-constrained perturbations and policy access to ψ | RARL · MuJoCo time-coupled worst case (Agg) | normalized performance: -0.86 ± 0.47 -> 2.54 ± 0.74 | Helped |
| Train M2TD3 with time-constrained perturbations and policy access to previous state and action | M2TD3 · MuJoCo static average case (Agg) | normalized performance: 1.0 ± 0.59 -> 2.23 ± 0.63 | Helped |
| Train RARL with time-constrained perturbations and state-only policy input | RARL · MuJoCo static worst case (Agg) | normalized performance: 0.33 ± 0.73 -> 0.92 ± 1.29 | No clear effect |
Surprising findings
- Oracle policies trained with the standard robust framework (Oracle RARL, Oracle M2TD3) were overly conservative, while the same oracle-conditioned policies trained with the TC framework performed much better.
Limitations
- Vanilla TC and Stacked TC lack convergence guarantees since the parameter is unobserved and the problem becomes partially observable.
- Theoretical guarantees are stated for the oracle variant; the Lipschitz bound for vanilla TC and Stacked TC relies on assumptions on reward and kernel variation.
- Evaluation is limited to MuJoCo continuous control with a 3D uncertainty set and a small perturbation radius of 0.001 for training.
Experimental setup
Datasets: MuJoCo
Benchmarks: MuJoCo Ant MuJoCo HalfCheetah MuJoCo Hopper MuJoCo Humanoid MuJoCo Walker
Metrics: normalized return worst-case return average return
Baselines: M2TD3 RARL Oracle M2TD3 Oracle RARL Domain randomization TD3
Builds on: RARL M2TD3 TD3 Robust value iteration
Closest papers by meaning
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Near-Optimal Distributionally Robust Reinforcement Learning with General $L_p$ Norms (NeurIPS 2024 · similarity 0.90)
- Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.90)
- Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees (NeurIPS 2025 · similarity 0.90)
- Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization (ICML 2025 · similarity 0.90)
- Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.