Research map / Reinforcement learning / Offline RL

On Rollouts in Model-Based Reinforcement Learning

ICLR 2025 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Infoprop, a model-based rollout mechanism that removes epistemic noise from ensemble predictions via an information-theoretic belief update and terminates rollouts by accumulated information loss, plugged into Dyna-style MBRL (Infoprop-Dyna) and evaluated on MuJoCo.

Problem

Model-based rollouts with trajectory sampling accumulate epistemic model error, distorting the synthetic data distribution and limiting useful rollout length in Dyna-style MBRL.

Approach

Separate aleatoric from epistemic uncertainty in an ensemble's predictive distribution, reconstruct an estimate of the environment state via a Kalman-style conditional update that discards epistemic noise, and quantify data corruption as accumulated conditional entropy to decide when to stop rollouts.

Key results

• Infoprop-Dyna performs on par with or better than MACURA and substantially outperforms MBPO in data efficiency and asymptotic performance on MuJoCo tasks.
• Infoprop-Dyna yields substantially increased average rollout lengths compared to prior methods in all environments except Ant, with rollouts up to 100 steps.
• Model data in Infoprop-Dyna closely follows the environment distribution on Hopper, while MBPO and MACURA show severe outliers.

Measured effects reported in the paper

Surprising findings

• Higher rollout data quality (smaller zeta_1) did not yield better learning: the most accurate setting (zeta_1=0.97) gave unstable returns, attributed to critic overfitting and loss of plasticity.
• Infoprop-Dyna shows instabilities on MuJoCo Humanoid late in training despite consistent rollout data.

Limitations

• Instabilities in learning on MuJoCo Humanoid when Infoprop rollouts are integrated naively into Dyna-style architecture.
• Assumptions (consistent aleatoric estimator, unbiased model) and heuristic thresholds lambda_1, lambda_2 are needed; thresholds depend on the AES model and algorithm.
• Rapid distribution shifts in the model buffer from long rollouts, especially early in training.

Experimental setup

Datasets: MuJoCo Halfcheetah MuJoCo Walker MuJoCo Hopper MuJoCo Ant MuJoCo Humanoid 1D random walk toy dataset

Benchmarks: MuJoCo Halfcheetah MuJoCo Walker MuJoCo Hopper MuJoCo Ant MuJoCo Humanoid

Metrics: episodic return average rollout length Wasserstein distance distribution consistency of model data vs environment data

Baselines: MBPO MACURA SAC Trajectory Sampling

Builds on: MBPO MACURA Trajectory Sampling Probabilistic ensembles Soft Actor-Critic

Closest papers by meaning

• Trust the Model Where It Trusts Itself - Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaption (ICML 2024 · similarity 0.89)
• Plan To Predict: Learning an Uncertainty-Foreseeing Model For Model-Based Reinforcement Learning (NeurIPS 2022 · similarity 0.89)
• Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · similarity 0.88)
• ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.88)
• Policy-shaped prediction: avoiding distractions in model-based reinforcement learning (NeurIPS 2024 · similarity 0.87)
• The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.87)