Research map / Probabilistic and scientific modeling / Variational and generative sampling
Power Mean Estimation in Stochastic Continuous Monte-Carlo Tree Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stochastic Power-HOOT, a continuous-action MCTS for stochastic MDPs combining a power-mean value backup with a polynomial HOO exploration bonus, with a claimed O(n^-zeta) convergence analysis and MuJoCo/control experiments.
Problem
Adapting MCTS to continuous action spaces and stochastic MDPs, where logarithmic exploration bonuses (HOOT) lack convergence guarantees under non-stationary value estimates and POLY-HOOT's guarantees only cover deterministic MDPs.
Approach
Combine a power-mean value backup operator with a polynomial exploration bonus inside HOO-based MCTS, extending POLY-HOOT's non-asymptotic convergence analysis from deterministic to stochastic continuous MDPs.
Key results
- On stochastic CartPole-IG, Stochastic Power-HOOT (p=2) reaches 77.85 ± 0.00 versus 68.46 for PW and 38.51 for HOOT.
- On stochastic Humanoid-v0, Power-HOOT (p=2) scores -44.12 ± 6.22, about 3.1x the UCT baseline, while HOOT scores -57.35.
- On stochastic MountainCar, Power-HOOT reports -27.285 ± 0.163 vs. -0.007 for HOOT, which is a large deviation from the other reported values and is not explained.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Stochastic Power-HOOT (p=2) instead of Discretized-UCT | MCTS · Pendulum | reward: 1263.68 \pm 37.66 -> 1397.40 \pm 95.42 | Helped |
| use Stochastic Power-HOOT (p=2) instead of HOOT | MCTS · MountainCar | reward: -0.007 \pm 0.001 -> 27.285 \pm 0.163 | Helped |
| use Stochastic Power-HOOT (p=2) instead of POLY-HOOT (p=1) | MCTS · Humanoid-v0 | reward: -44.40 \pm 3.33 -> -44.12 \pm 6.22 | Helped |
| use Stochastic Power-HOOT (p=8) instead of POLY-HOOT (p=1) | MCTS · Hopper-v0 | reward: 13230.66 \pm 2844.33 -> 13348.45 \pm 6110.36 | Helped |
| set power parameter p to 2 instead of 1 (ablation) | Stochastic Power-HOOT · CartPole-IG | reward: 76.91 -> 77.85 | Helped |
| set power parameter p to 3 instead of 2 (ablation) | Stochastic Power-HOOT · CartPole-IG | reward: 77.85 -> 76.80 | Hurt |
| set power parameter p to 4 instead of 2 (ablation) | Stochastic Power-HOOT · CartPole-IG | reward: 77.85 -> 77.77 | No clear effect |
| vary power parameter p from 5 through 10 instead of using p=2 (ablation) | Stochastic Power-HOOT · CartPole-IG | reward: 77.85; 76.81; 75.85; 75.75; 76.74; 75.82; 75.89 | Hurt |
| set power parameter p to 8 instead of 2 (ablation) | Stochastic Power-HOOT · Humanoid-v0 | reward: -44.12 \pm 6.22 -> -44.40 \pm 3.33 | No clear effect |
| set power parameter p to 2 instead of 8 (ablation) | Stochastic Power-HOOT · Hopper-v0 | reward: 13348.45 \pm 6110.36 -> 13303.61 \pm 3070.34 | No clear effect |
Surprising findings
- Voronoi MCTS, a deterministic-setting method, is reported 12.7x worse than UCT on Hopper, the paper's main evidence for stochastic robustness.
- The reported MountainCar score for Power-HOOT (-27.285) is far outside the range of all other methods' scores (about -0.007 to -0.055), with no explanation.
Limitations
- The convergence proof is a sketch relying on the non-stationary bandit result of prior work and an inductive argument that the paper itself describes as partly borrowed.
- Experiments use custom stochastic variants of standard benchmarks with ad hoc reward transformations and report results with no code release.
- Theorem 4's proof is only sketched in the main text; the appendix proofs are said to follow prior work with a missing concentration bound added.
Experimental setup
Datasets: CartPole CartPole-IG Pendulum MountainCar Acrobot Humanoid-v0 Hopper-v0
Benchmarks: CartPole CartPole-IG Pendulum MountainCar Acrobot Humanoid-v0 Hopper-v0
Metrics: episode reward (mean ± std)
Baselines: UCT Discretized-UCT PW (progressive widening) HOOT POLY-HOOT Voronoi MCTS
Builds on: HOOT POLY-HOOT Hierarchical Optimistic Optimization UCT
Closest papers by meaning
- Graph-Based Stochastic-Power-UCT: Monte-Carlo Graph Search with Power Mean Estimation (NeurIPS 2026 · similarity 0.93)
- Monte-Carlo Tree Search with Uncertainty Propagation via Optimal Transport (ICML 2025 · similarity 0.89)
- Online Robust Reinforcement Learning Through Monte-Carlo Planning (ICML 2025 · similarity 0.89)
- Monte Carlo Tree Search in the Presence of Transition Uncertainty (AAAI 2024 · similarity 0.89)
- Threshold UCT: Cost-Constrained Monte Carlo Tree Search with Pareto Curves (AAAI 2025 · similarity 0.88)
- Graph Sparse Sampling: Breaking the Curse of the Horizon in Continuous MDP Planning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.