Research map / Reinforcement learning / Visual and representation RL
SOMBRL: Scalable and Optimistic Model-Based RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOMBRL, which greedily maximizes extrinsic reward plus weighted epistemic uncertainty in model-based RL, giving sublinear regret guarantees across finite-horizon, discounted, and nonepisodic settings while scaling to visual control and a hardware RC car.
Problem
Efficient exploration in model-based RL with unknown dynamics: principled optimistic exploration has strong theory but is computationally prohibitive in high-dimensional settings, while practical MBRL relies on naive exploration.
Approach
Instead of optimizing over plausible models or sampling dynamics, plan with the mean dynamics and add the model's epistemic uncertainty as an intrinsic reward weighted by lambda_n; this is shown to yield an optimistic upper bound on true return and sublinear regret, while remaining compatible with any planner or policy optimizer.
Key results
- On the RC car hardware with sparse rewards, SimFSVGD-Optimistic learns the parking maneuver while SimFSVGD succeeded only once in 5 attempts.
- In state-based DMC tasks, MBPO-Optimistic achieves the best performance across tasks, solving sparse-reward MountainCar and CartPole where greedy baselines fail.
- Dreamer-Optimistic matches or beats Dreamer across visual tasks and solves Atari Venture where Dreamer obtains zero reward; HUCRL requires about 3x more compute than SOMBRL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add epistemic-uncertainty intrinsic reward to MBPO | MBPO · DMC state-based tasks, including MountainCar and CartPole | episodic return | Helped |
| remove the epistemic-uncertainty intrinsic reward (ablation) | MBPO-Optimistic · DMC state-based tasks | episodic return | Hurt |
| add epistemic-uncertainty intrinsic reward to Dreamer | Dreamer · DMC Finger-spin | episodic return | Helped |
| add epistemic-uncertainty intrinsic reward to Dreamer | Dreamer · Atari Venture | episodic return | Helped |
| remove the epistemic-uncertainty intrinsic reward (ablation) | Dreamer-Optimistic · DMC and Atari visual-control tasks | episodic return | Mixed |
| add epistemic-uncertainty intrinsic reward to Dreamer | Dreamer · DMC Finger-spin and CartPole with action costs | episodic return | Helped |
| add epistemic-uncertainty intrinsic reward to SimFSVGD | SimFSVGD · RC-car simulation, dense reward | task performance | No clear effect |
| add epistemic-uncertainty intrinsic reward to SimFSVGD | SimFSVGD · RC-car simulation, sparse rewards | task performance | Helped |
| make the reward sparser | SimFSVGD · RC-car simulation | task performance | Hurt |
| add epistemic-uncertainty intrinsic reward to SimFSVGD | SimFSVGD · RC-car hardware, sparse reward | task success | Helped |
| use HUCRL's optimistic exploration instead of OMBRL | GP dynamics experiments · Pendulum and MountainCar, episodic | episodic return | Hurt |
| use HUCRL instead of OMBRL | GP dynamics experiments · Pendulum and MountainCar | compute time: roughly 3× more compute time | Hurt |
Surprising findings
- Dreamer fails to solve tasks with action costs on Finger spin and CartPole, while the optimistic variant explores successfully.
Limitations
- Requires training a probabilistic model (e.g., deep ensembles) to quantify epistemic uncertainty.
- Theory assumes GP models in RKHS with known process noise and continuous dynamics; deep models are only empirically justified.
- The lambda_n hyperparameter is set heuristically via auto-tuning in practice rather than from theory.
Experimental setup
Datasets: DeepMind Control Suite Atari OpenAI Gym Pendulum OpenAI Gym MountainCar RC car hardware data
Benchmarks: DeepMind Control Suite Atari OpenAI Gym Pendulum OpenAI Gym MountainCar RC car parking (simulation and hardware)
Metrics: episodic return cumulative regret average reward
Baselines: MBPO-Mean MBPO-PETS Dreamer SAC MaxInfoRL HUCRL PETS SimFSVGD
Builds on: MBPO Dreamer SimFSVGD PETS HUCRL
Closest papers by meaning
- NeoRL: Efficient Exploration for Nonepisodic RL (NeurIPS 2024 · similarity 0.92)
- Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling (ICLR 2025 · similarity 0.90)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL (NeurIPS 2025 · similarity 0.89)
- Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds (ICLR 2025 · similarity 0.88)
- MetaCURL: Non-stationary Concave Utility Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.