Research map / Probabilistic and scientific modeling / Molecules and proteins
Counterfactual Online Learning for Open-Loop Monte-Carlo Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CORAL, an open-loop MCTS that uses causal multi-armed bandits with unobserved confounders (MABUC) and human-preferred actions to learn observational and counterfactual values online, achieving near closed-loop performance with much smaller trees.
Problem
Open-loop MCTS for POMDP online planning is memory-efficient but generally underperforms closed-loop MCTS, even when human knowledge (preferred actions) is integrated.
Approach
Two online learning phases: first learn observational values for preferred actions (intent policy), then learn counterfactual values with MABUCs conditioned on the intent, so states are considered indirectly through the intent policy without storing them in the tree.
Key results
- CORAL is among the best open-loop approaches when simulation budget exceeds 1000 and reaches performance similar to closed-loop planning in the four POMDP benchmarks.
- Standard open-loop MCTS with preferred actions keeps up with CORAL only in RockSample[11,11] and RockSample[15,15].
- CORAL achieves competitive performance with closed-loop algorithms while constructing significantly smaller search trees (only slightly larger than standard open-loop).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CORAL instead of standard open-loop MCTS with legal actions | Open-loop MCTS · Four POMDP benchmark scenarios | performance | Helped |
| Use CORAL instead of standard open-loop MCTS with preferred actions only | Open-loop MCTS · Four POMDP benchmark scenarios | performance | Helped |
| Use CORAL instead of closed-loop planning | MCTS planning · Four POMDP benchmark scenarios | tree size | Helped |
| Use CORAL instead of stack-based planning | Open-loop planning · Four POMDP benchmark scenarios | performance | Helped |
| Vary the training ratio among 25%, 50%, and 75% (ablation) | CORAL · Four POMDP benchmark scenarios | performance | No clear effect |
| Remove preferred-action human knowledge by setting Apref = Alegal (ablation) | CORAL · Four POMDP benchmark scenarios | performance | Hurt |
| Set η = 0, removing the human-knowledge training phase (ablation) | CORAL · Four POMDP benchmark scenarios | performance | Hurt |
| Set η = 1, relying exclusively on preferred actions (ablation) | CORAL · Four POMDP benchmark scenarios | performance | Hurt |
| Use lazy initialization for counterfactual bandits (ablation) | CORAL · Four POMDP benchmark scenarios | tree size | Helped |
Surprising findings
- Open-loop MCTS with preferred actions performs better than CORAL at low budgets and matches it in some RockSample scenarios.
- Without human knowledge (Apref = Alegal), CORAL performs noticeably worse, indicating preferred actions must capture problem structure.
Limitations
- Requires a sufficient simulation budget (nb > 1000) for the intent policy to be meaningful.
- Depends on human-provided preferred actions that must capture structural properties of the problem.
- Worst-case node count can grow quadratically with the number of actions.
Experimental setup
Benchmarks: Tag RockSample[11,11] RockSample[15,15] PocMan
Metrics: average return number of nodes created (inverse) vs performance
Baselines: POMCP POMCPOW AR-DESPOT open-loop MCTS stack-based planning
Builds on: POMCP open-loop MCTS (Phan et al. 2019a) MABUC (Bareinboim, Forney, and Pearl 2015) Thompson Sampling
Closest papers by meaning
- Threshold UCT: Cost-Constrained Monte Carlo Tree Search with Pareto Curves (AAAI 2025 · similarity 0.85)
- Online Robust Reinforcement Learning Through Monte-Carlo Planning (ICML 2025 · similarity 0.84)
- Monte Carlo Tree Search in the Presence of Transition Uncertainty (AAAI 2024 · similarity 0.84)
- Power Mean Estimation in Stochastic Continuous Monte-Carlo Tree Search (ICML 2025 · similarity 0.84)
- A Bayesian Approach to Online Planning (ICML 2024 · similarity 0.84)
- Exploring Starts Are Not Enough: Counterexamples and a Fix for Monte Carlo Exploring Starts (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.