Research map / Reinforcement learning / Visual and representation RL
HarmonyDream: Task Harmonization Inside World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HarmonyDream, which learns adaptive (rectified) loss weights for observation, reward, and dynamics losses in Dreamer-style explicit world models to balance observation and reward modeling, improving sample efficiency on visual control and Atari 100K.
Problem
Explicit model-based RL world models are dominated by observation reconstruction losses, whose scale is orders of magnitude larger than the reward loss, which hurts sample efficiency; default equal loss weights ignore this imbalance.
Approach
Treat world model learning as multi-task learning and attach learnable harmonizers (variational uncertainty-style weights with a rectified log(1+sigma) term) to each loss so the losses are dynamically rescaled to a harmonized scale, adding no extra hyperparameters.
Key results
- HarmonyDream on DreamerV2 gains 10%-69% absolute performance boosts on visual robotic tasks, with up to 90% more success on Meta-world Assembly.
- Harmony DreamerV3 reaches 136.5% mean human normalized score on Atari 100K, a new state of the art among methods without lookahead search.
- HarmonyDream adds no extra hyperparameters, only one learnable parameter per loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the reward loss coefficient (ablation) | DreamerV2 · Meta-world tasks | sample efficiency | Helped |
| Remove the observation loss (w_o=0), leaving reward modeling alone (ablation) | DreamerV2 · Meta-world analysis tasks | sample efficiency | Hurt |
| Balance observation and reward modeling with HarmonyDream | DreamerV2 · Meta-world robotic manipulation | success rate: 10%−69% higher success rates or episode returns | Helped |
| Balance observation and reward modeling with HarmonyDream | DreamerV2 · RLBench Push Button | final performance: 28% | Helped |
| Balance observation and reward modeling with HarmonyDream | DreamerV2 · RLBench Reach Target | final performance: 50% | Helped |
| Balance world-model task losses with HarmonyDream | DreamerV3 · Atari 100K | mean human performance: 136.5% | Helped |
| Replace the rectified harmonious loss with its unrectified version (ablation) | HarmonyDream · world-model training | training stability | Hurt |
| Use HarmonyDream instead of DreamerV2 | DreamerV2 · DMC Remastered | dynamics loss convergence | Helped |
| Use HarmonyDream instead of DreamerV3 | DreamerV3 · Meta-world and RLBench | performance | Helped |
| Use HarmonyDream instead of DreamerPro's manually tuned loss weights | DreamerPro · DMC Remastered | sample efficiency | Helped |
Surprising findings
- Simply increasing the reward loss weight (w_r = 10 or 100) dramatically improves DreamerV2 sample efficiency on Meta-world.
- Pure implicit (reward-only) world model learning with w_o=0 yields inferior, high-variance results compared with balanced explicit learning.
Limitations
- Method is primarily empirical and intuitive, lacking theoretical explanation beyond loss scales.
- Benefits are marginal on clean benchmarks such as standard DMC and Crafter, where observation domination is not significant.
- Evaluated on a subset of Meta-world tasks and only two RLBench tasks due to compute and task difficulty.
Experimental setup
Datasets: Meta-world RLBench DMC Remastered Atari 100K MineDojo Hunt Cow Natural background DMC
Benchmarks: Meta-world RLBench DMC Remastered Atari 100K MineDojo Hunt Cow Natural background DMC
Metrics: success rate episode return human normalized score (mean and median) 95% confidence interval over runs
Baselines: DreamerV2 DreamerV3 DreamerPro TD-MPC Denoised MDP RePo Uncertainty Weighting Dynamics Weight Average NashMTL
Builds on: DreamerV2 DreamerV3 DreamerPro uncertainty weighting
Closest papers by meaning
- HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning (ICML 2024 · similarity 0.87)
- PWM: Policy Learning with Multi-Task World Models (ICLR 2025 · similarity 0.87)
- SELF-HARMONY: LEARNING TO HARMONIZE SELF-SUPERVISION AND SELF-PLAY IN TEST-TIME REINFORCEMENT LEARNING (ICLR 2026 · similarity 0.87)
- DyMoDreamer: World Modeling with Dynamic Modulation (NeurIPS 2025 · similarity 0.87)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Meta-World+: An Improved, Standardized, RL Benchmark (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.