Research map / Reinforcement learning / MDPs and planning

Stop! Planner Time: Metareasoning for Probabilistic Planning Using Learned Performance Profiles

AAAI 2024 · Reinforcement learning › MDPs and planning · approach: Average-reward and regularized MDPs · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a non-myopic metareasoning method for anytime probabilistic (SSP MDP) planners that learns contextual performance profiles and uses DQN to decide when to stop planning and which planner hyperparameter to use.

Problem

Metalevel control of anytime probabilistic planning with MDPs, where the current solution quality is not known exactly and planning cost must be traded off against execution cost.

Approach

Abstracts the metalevel MDP to hand-designed planner features plus a problem context vector, dropping the assumption that current solution quality is observable; a deep RL agent learns the cost function from experience, implicitly learning contextual performance profiles, and picks planner hyperparameters (WBRTDP heuristic weight) alongside stop/continue actions.

Key results

• In the deep sea treasure domain, the proposed method achieves the lowest normalised thinking-plus-acting cost on average, significantly better (p=0.01, one-sided Mann-Whitney U) than all ablations and the MidBound and PolicyEval alternatives.
• In the race track domain, the method is significantly better than all alternatives except the NoFeatures ablation, which performs almost identically.
• PolicyEval's thinking cost rose by around 50% to 100%, causing it to terminate very early and perform poorly.

Measured effects reported in the paper

Surprising findings

• Evaluating the current policy at every metalevel step (PolicyEval) is too expensive to be offset by better solution quality estimates, so it performs worse than cheaper estimates.
• Algorithm features are not particularly useful in the race track domain; the NoFeatures ablation performs almost identically to the full method.

Limitations

• Abstraction of algorithm configuration to features may yield a non-Markov state transition function.
• Assumes a default object-level policy and single-shot rather than online planning.
• Evaluated only on two synthetic domains with a single object-level planner (WBRTDP).

Experimental setup

Datasets: randomly generated deep sea treasure problem instances randomly generated race track problem instances

Benchmarks: deep sea treasure (SSP variant) race track

Metrics: normalised combined thinking and acting cost metalevel episode length

Baselines: MidBound PolicyEval NoFeatures ablation NoContext ablation NoTuning ablation

Builds on: Bhatia et al. 2022 metareasoning with hyperparameter tuning Sung et al. 2021 learned metareasoning for RRT* BRTDP DQN Lin et al. 2015 metamyopic SSP metareasoning

Closest papers by meaning

• A Formal Metareasoning Model of Concurrent Planning and Execution (AAAI 2023 · similarity 0.88)
• Metareasoning in Modular Software Systems: On-the-Fly Configuration Using Reinforcement Learning with Rich Contextual Representations (AAAI 2020 · similarity 0.86)
• TodoEvolve: Learning to Architect Agent Planning Systems (ICML 2026 · similarity 0.85)
• Meta Reinforcement Learning with Finite Training Tasks - a Density Estimation Approach (NeurIPS 2022 · similarity 0.85)
• Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
• Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modeling (ICML 2025 · similarity 0.84)