Research map / Reinforcement learning / Policy gradient and actor-critic
Trust the Model Where It Trusts Itself - Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MACURA, a Dyna-style MBPO variant that adapts branched rollout length per state using a closed-form geometric Jensen-Shannon uncertainty of a probabilistic ensemble, with a self-tuning threshold, and reports gains on MuJoCo.
Problem
In Dyna-style model-based RL, choosing branched rollout lengths is hard; global schedules assume uniform model accuracy over training, which fails for complex systems and extrapolation.
Approach
Treat model accuracy as a local property: stop model rollouts where the probabilistic ensemble's pairwise geometric Jensen-Shannon uncertainty exceeds an adaptive threshold kappa, derived from a monotonic-improvement bound restricted to a state subset E; also scale gradient updates with model buffer fill.
Key results
- MACURA learns substantially faster than MBPO and M2AC, especially on Humanoid and Ant, and shows stronger asymptotic performance than other model-based approaches.
- MACURA frequently matches or outperforms the model-free SAC baseline in asymptotic performance.
- Fixed T_max=10 and zeta=95% are used across all tasks; only the scaling factor xi needs task-specific tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use uncertainty-aware adaptive rollout lengths | MACURA · MuJoCo benchmark | data efficiency | Helped |
| use uncertainty-aware adaptive rollout lengths | MACURA · MuJoCo benchmark | asymptotic performance | Helped |
| use model-based rollouts | MBPO, M2AC, and MACURA · MuJoCo benchmark | learning speed | Helped |
| use pink noise exploration instead of deterministic or white noise exploration (ablation) | MACURA and MBPO · Humanoid | performance and learning stability | Helped |
| replace threshold-based rollout adaptation with rank-based rollout adaptation (ablation) | MACURA · Humanoid | performance | Hurt |
| use a reliable GJS uncertainty estimate instead of One-vs-Rest uncertainty (ablation) | MACURA · Humanoid | learning stability | Helped |
| adapt the uncertainty threshold instead of fixing it at 300 (ablation) | MACURA · Humanoid | learning stability: 300 | Helped |
| adapt the uncertainty threshold instead of fixing it at 150 (ablation) | MACURA · Humanoid | learning stability: 150 | Helped |
| adapt gradient steps to the model-buffer data amount instead of fixing gradient steps at G=20 (ablation) | MACURA · Humanoid | performance: G=20 | Helped |
| use pink noise exploration instead of deterministic environment interaction (ablation) | MACURA · Humanoid | performance | Helped |
| use an intermediate scaling factor ξ instead of a very high or very low value (ablation) | MACURA · Humanoid | learning stability | Helped |
Surprising findings
- Low values of the scaling factor xi also destabilize training, attributed to narrow model-buffer distributions causing Q-function overfitting.
- Replacing the GJS uncertainty with the M2AC one-vs-rest uncertainty leads to divergence on Humanoid.
Limitations
- Monotonic improvement bound relies on approximating the stopping time by the model-side stopping time, which is assumed not to diverge much in practice.
- Requires tuning the scaling factor xi, and both too high and too low values destabilize learning.
- Evaluated only on MuJoCo state-based tasks and a toy pendulum.
Experimental setup
Datasets: MuJoCo environment interaction data
Benchmarks: MuJoCo
Metrics: expected return data efficiency
Baselines: MBPO M2AC SAC
Builds on: MBPO Soft Actor-Critic probabilistic ensemble M2AC mbrl-lib
Closest papers by meaning
- On Rollouts in Model-Based Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Policy-shaped prediction: avoiding distractions in model-based reinforcement learning (NeurIPS 2024 · similarity 0.86)
- Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · similarity 0.86)
- ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.85)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.