Research map / Reinforcement learning / Policy gradient and actor-critic
Online Robust Reinforcement Learning Through Monte-Carlo Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a robust MCTS variant (Robust Power-UCT) that uses a power-mean backup and optimistic exploration bonuses with TV, chi-squared, and Wasserstein ambiguity sets, with finite-sample O(n^{-1/2}) root value convergence and experiments on Gambler's, Frozen Lake, and American option pricing.
Problem
Monte Carlo Tree Search assumes the simulator matches real dynamics, so planning degrades under simulation-to-reality model mismatch; no MCTS method previously handled robust (ambiguous) transition and reward models with theory.
Approach
Embed distributionally robust (worst-case) transition and reward backups into MCTS via a power-mean backup operator and carefully designed optimistic exploration bonuses, analyzed as non-stationary bandits, yielding finite-sample convergence comparable to standard MCTS.
Key results
- On Frozen Lake with p_slip plan 0.3 and execution 0.1, Wasserstein robust Power-UCT reaches 58% success vs 15% for Stochastic-Power-UCT.
- Robust variants (Chi-squared, Wasserstein) maintain stable American option returns across execution probabilities where standard Power-UCT drops from about 5 to near 0 at p_u=0.8.
- Theory shows the root value estimate converges at O(n^{-1/2}), matching standard MCTS rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a Wasserstein ambiguity set instead of Stochastic-Power-UCT | Robust-Power-UCT · Frozen Lake, plan slip=0.3 and exec slip=0.1 | success rate (%): 15 -> 58 | Helped |
| Use Chi-squared ambiguity set instead of Stochastic-Power-UCT | Robust-Power-UCT · Frozen Lake, plan slip=0.3 and exec slip=0.3 | success rate (%): 10 -> 35 | Helped |
| Use Wasserstein ambiguity set instead of Stochastic-Power-UCT | Robust-Power-UCT · Frozen Lake, plan slip=0.4 and exec slip=0.4 | success rate (%): 5 -> 20 | Helped |
| Use Wasserstein ambiguity set instead of Total Variation ambiguity set | Robust-Power-UCT · Frozen Lake, plan slip=0.3 | success rate (%): 18 -> 58; 12 -> 32 | Mixed |
| Account for transition ambiguity with Wasserstein or Chi-squared sets instead of the baseline | Robust-Power-UCT · Frozen Lake | success rate (%) | Helped |
Surprising findings
- Robust variants also outperform the baseline when planning and execution probabilities match in Frozen Lake, suggesting conservatism helps even without model mismatch.
Limitations
- Theoretical dependence on number of actions, states, and depth is not explicit, so not directly comparable to other robust RL bounds.
- Formulation is acknowledged to be overly conservative.
- Experiments only vary transition ambiguity in the main text, not reward ambiguity.
Experimental setup
Benchmarks: Gambler's Problem Frozen Lake American Option Pricing
Metrics: success rate winning ratio average return
Baselines: Stochastic-Power-UCT
Builds on: Power-UCT Stochastic-Power-UCT robust MDP UCT
Closest papers by meaning
- Monte Carlo Tree Search in the Presence of Transition Uncertainty (AAAI 2024 · similarity 0.90)
- Power Mean Estimation in Stochastic Continuous Monte-Carlo Tree Search (ICML 2025 · similarity 0.89)
- MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planning (NeurIPS 2025 · similarity 0.88)
- Threshold UCT: Cost-Constrained Monte Carlo Tree Search with Pareto Curves (AAAI 2025 · similarity 0.87)
- Monte-Carlo Tree Search with Uncertainty Propagation via Optimal Transport (ICML 2025 · similarity 0.87)
- Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.