Research map / Reinforcement learning / Visual and representation RL
Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOT-GP, a model-based RL method that uses a GP with neural-network mean over joint state-reward dynamics, sampling optimistic rewards from a truncated distribution and then the expected next state for optimistic Thompson-style exploration.
Problem
Sample-efficient exploration in model-based RL for continuous control when rewards are unknown and sparse, requiring optimism informed by joint uncertainty over transitions and rewards.
Approach
Model state and reward jointly with a GP (neural-network mean, Kronecker/coregionalized covariance), then sample an optimistic reward from a truncated distribution above a threshold r_min and set the next state to the expected transition conditioned on that reward, a two-step adaptation of Thompson sampling that ties optimism over reward to plausible states.
Key results
- On Half-Cheetah, HOT-GP with r_min 0.1 to 0.3 reaches SAC's final performance in 250,000 environment steps versus about 3 million for SAC.
- On the Medium Maze sparse navigation task, HOT-GP variants deliver the strongest performance while MBPO and optimistic MBPO learn less efficiently.
- On the Coverage task, HOT-GP achieves strong performance within 200,000 environment steps and marginally exceeds DDPG's best performance, outperforming H-UCRL with and without ground-truth reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HOT-GP instead of MBPO or optimistic MBPO | Model-based exploration · Sparse Reacher | sample efficiency | Helped |
| Use HOT-GP instead of greedy exploitation | Model-based exploration · Pusher | sample efficiency | Helped |
| Use HOT-GP with r_min ranging from 0.1 to 0.7 | HOT-GP · Medium Maze | performance | Helped |
| Use HOT-GP instead of H-UCRL | Optimistic exploration · Coverage | performance | Helped |
| Increase r_min range from 0.1 to 0.5 (ablation) | HOT-GP · Coverage | performance | Helped |
| Remove k-branching (ablation) | HOT-GP · Coverage | performance | Hurt |
| Set k-branching to k=5 instead of k=1 (ablation) | HOT-GP · Coverage | performance | Hurt |
| Remove correlation modeling between predicted state and reward (ablation) | HOT-GP · Coverage | performance | Hurt |
| Sample the next state instead of selecting its conditional expectation (ablation) | HOT-GP · Coverage | performance variance | Hurt |
| Remove uncertainty clamping (ablation) | MBPO · Half-Cheetah | learning performance | Hurt |
| Use deterministic greedy predictions rather than uncertainty-aware methods | Model-based exploration · Half-Cheetah | learning speed | Hurt |
| Use deterministic greedy predictions rather than uncertainty-aware methods | Model-based exploration · Pusher | performance | No clear effect |
Surprising findings
- Removing log-variance clamping from MBPO causes a complete failure to learn, indicating poorly calibrated model uncertainty without clamping.
- Greedy exploitation with a deterministic model slows learning on Half-Cheetah but shows no impact on Pusher, while optimistic exploration helps more on Pusher than uncertainty-aware MBPO alone.
Limitations
- Higher wall-clock cost due to GP training compared with MBPO and greedy baselines.
- Only a linear schedule for r_min was considered and the optimism hyperparameter is domain dependent.
- Evaluated on six tasks only and only one-step lookahead model rollouts (K=1 best); the H-UCRL approximation and Thompson sampling show instability in some settings.
Experimental setup
Datasets: MuJoCo Half-Cheetah, Reacher, Pusher Sparse Reacher (custom) D4RL sparse maze (U Maze, Medium Maze) VMAS Coverage (custom)
Benchmarks: MuJoCo Half-Cheetah MuJoCo Reacher MuJoCo Pusher Sparse Reacher D4RL U Maze D4RL Medium Maze VMAS Coverage
Metrics: average evaluation episode reward sample efficiency (environment steps to convergence) wall clock runtime
Baselines: SAC MBPO Optimistic MBPO Thompson sampling Greedy exploitation H-UCRL DDPG PPO
Builds on: H-UCRL MBPO PETS SAC GP-UCRL
Closest papers by meaning
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.90)
- SOMBRL: Scalable and Optimistic Model-Based RL (NeurIPS 2025 · similarity 0.90)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environments (AAAI 2024 · similarity 0.88)
- SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Minimax Optimal Reinforcement Learning with Quasi-Optimism (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.