Research map / Reinforcement learning / Visual and representation RL
DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamSmooth, which trains MBRL reward models on temporally smoothed rewards (Gaussian, uniform, or EMA) to ease sparse reward prediction, improving DreamerV3, TD-MPC and MBPO on sparse-reward tasks.
Problem
Reward models in model-based RL struggle to predict sparse rewards at exact timesteps, especially under partial observability or stochastic dynamics, and this reward prediction error bottlenecks policy learning.
Approach
Temporally smooth the rewards of each collected episode with a sum-preserving kernel (Gaussian, uniform, or EMA) before storing them in the replay buffer, so the reward model only needs to predict approximate reward timing; the agent learns only from smoothed rewards. Requires one added line of code.
Key results
- DreamSmooth-Gaussian and -Uniform significantly improve DreamerV3 sample efficiency and final performance on RoboDesk, Hand, and Earthmoving sparse-reward tasks.
- Reward prediction on Crafter achievements is better with reward smoothing in 15 of 19 achievements.
- DreamSmooth enables TD-MPC and MBPO to complete the first Hand task, where vanilla versions fail; smoothing is neutral on DMC and Atari.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Gaussian reward smoothing | DreamerV3 · RoboDesk | task performance | Helped |
| Apply uniform reward smoothing | DreamerV3 · RoboDesk | task performance | Helped |
| Apply Gaussian or uniform reward smoothing | DreamerV3 · Hand | task performance | Helped |
| Apply Gaussian or uniform reward smoothing | DreamerV3 · Earthmoving | task performance | Helped |
| Apply Gaussian or uniform reward smoothing | DreamerV3 · Crafter | task performance | Hurt |
| Apply EMA reward smoothing | DreamerV3 · Crafter | task performance | Helped |
| Apply reward smoothing | DreamerV3 · DMC and Atari | task performance | No clear effect |
| Apply reward smoothing | TD-MPC · Hand | task performance | Helped |
| Apply reward smoothing | MBPO · Hand | task performance | Helped |
| Apply reward smoothing | MBPO · DMC dense-reward tasks | task performance | No clear effect |
| Oversample sequences containing sparse rewards (ablation) | DreamerV3 · RoboDesk | task performance | Helped |
| Increase reward model size without reward smoothing (ablation) | DreamerV3 · RoboDesk and Hand | task performance | No clear effect |
Surprising findings
- On Crafter, Gaussian and uniform smoothing improve reward prediction but degrade task performance versus vanilla DreamerV3, attributed to false-positive reward predictions from anticipating future rewards.
- Increasing reward model size has negligible effect on reward prediction without smoothing.
Limitations
- No theoretical optimality guarantee for symmetric (Gaussian/uniform) smoothing that uses future rewards, since the smoothed reward is policy-dependent.
- Improved reward prediction does not always improve task performance; Crafter performance drops with Gaussian and uniform smoothing.
- MBPO on Hand and RoboDesk required replay buffer initialization from fully-trained DreamerV3 demonstrations.
Experimental setup
Datasets: RoboDesk (modified) Shadow Hand (modified) Earthmoving (AGX simulator) Crafter DeepMind Control Suite (7 tasks) Atari (6 tasks at 100K steps)
Benchmarks: RoboDesk Hand Earthmoving Crafter DeepMind Control Suite Atari 100K
Metrics: episode return / task success over environment steps reward prediction rate for Crafter achievements
Baselines: DreamerV3 TD-MPC MBPO DreamerV3 with oversampling of sparse rewards
Builds on: DreamerV3 TD-MPC MBPO reward shaping (potential-based)
Closest papers by meaning
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.87)
- ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization (ICLR 2025 · similarity 0.87)
- Learning Process Rewards via Visitation Matching for Efficient RL (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.