Research map / Reinforcement learning / MDPs and planning
Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernel
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EWoK, which makes any off-the-shelf non-robust RL algorithm robust in KL-uncertainty RMDPs by resampling next states from the nominal kernel with weights that approximate the worst kernel.
Problem
Learning policies robust to transition perturbations in robust MDPs is hard to scale to high-dimensional domains, since existing model-free robust RL methods rely on regularizing particular RL algorithms and model-based RMDP solvers need the nominal kernel in tabular settings.
Approach
Instead of regularizing policy or value updates, the method characterizes the worst kernel for KL uncertainty sets as a reweighting of the nominal kernel (P_worst ∝ P_nominal · exp(-δ)), and approximates it by drawing N next-state samples from the nominal simulator and resampling one with probability proportional to exp(-(v(s')-mean v)/κ), then training any non-robust RL algorithm on the resulting
Key results
- On CartPole, EWoK outperforms the baseline DDQN and PPO and domain randomization when noise and environment parameters deviate from nominal values (IQM with CIs, 40 seeds DDQN, 20 seeds PPO).
- On DeepMind Control walker tasks with SAC and TD3, EWoK suffers less performance degradation as action-noise mean deviates from zero, clearly outperforming the non-robust baseline, though on walker-run it gets lower nominal reward (robustness-performance trade-off).
- Theory: the estimated kernel's value iteration converges at rate gamma^n in sup-norm to the robust value function (Theorem on kernel convergence).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EWoK to DDQN | DDQN · Cartpole, noise perturbations | reward | Helped |
| Add EWoK to PPO | PPO · Cartpole, noise perturbations | reward | Helped |
| Add EWoK to DDQN or PPO | DDQN or PPO · Cartpole, environment-parameter perturbations | reward | Helped |
| Add EWoK to SAC or TD3 | SAC or TD3 · DeepMind Control Suite, noise perturbations | reward | Helped |
| Add EWoK to SAC or TD3 | SAC or TD3 · DeepMind Control Suite, environment-parameter perturbations | reward | Helped |
| Add EWoK to SAC or TD3 | SAC or TD3 · Walker-run, nominal and perturbed dynamics | reward | Mixed |
| Decrease κ (ablation) | EWoK · Walker tasks, noise perturbations | reward | Helped |
| Decrease κ too much (ablation) | EWoK · DeepMind Control Suite | reward | Hurt |
| Use a large κ instead of a smaller κ (ablation) | EWoK · Walker-run, nominal and perturbed dynamics | reward | Mixed |
| Use a small sample size N (ablation) | EWoK · DeepMind Control Suite, noise perturbations | reward difference versus baseline | Hurt |
| Increase sample size N (ablation) | EWoK · DeepMind Control Suite, noise perturbations | reward difference versus baseline | Mixed |
Surprising findings
- On walker-run, EWoK achieves lower reward under the nominal dynamics but better reward under perturbations, showing a robustness-performance trade-off.
- Larger kappa (less robustness) can achieve high nominal performance but significantly underperforms under perturbation; decreasing kappa too much can lead to overly conservative policies.
Limitations
- Requires the ability to sample next states from the transition model multiple times per step, which may be slow in some environments.
- Robustness-performance trade-off: worst-case optimization can reduce nominal performance (e.g., walker-run), and the sensitivity to kappa is environment-dependent.
- Theory covers KL uncertainty sets, while experiments also use non-KL and coupled perturbations without theoretical guarantees.
Experimental setup
Datasets: OpenAI Gym CartPole OpenAI Gym CliffWalking DeepMind Control Suite walker-stand DeepMind Control Suite walker-walk DeepMind Control Suite walker-run Real-World RL Suite perturbations
Benchmarks: CartPole with noise and environment parameter perturbations DeepMind Control walker-stand, walker-walk, walker-run under action noise and joint damping/thigh/torso perturbations CliffWalking tabular toy task
Metrics: interquartile mean (IQM) return 95% stratified bootstrap confidence intervals
Baselines: Double DQN PPO SAC TD3 domain randomization
Builds on: robust MDP theory (sa-rectangularity) Double DQN PPO SAC TD3
Closest papers by meaning
- Online MDP with Prototypes Information: A Robust Adaptive Approach (AAAI 2025 · similarity 0.90)
- Policy Optimization for Robust Average Reward MDPs (NeurIPS 2024 · similarity 0.90)
- A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processes (ICML 2024 · similarity 0.89)
- Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization (ICML 2025 · similarity 0.89)
- Provable Policy Gradient for Robust Average-Reward MDPs Beyond Rectangularity (ICML 2025 · similarity 0.88)
- Efficient Value Iteration for s-rectangular Robust Markov Decision Processes (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.