Research map / Reinforcement learning / Policy gradient and actor-critic

MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MAD-TD, which replaces 5% of critic batch samples with on-policy transitions generated by a learned latent world model, stabilizing high update-ratio TD3-style RL on hard DeepMind Control tasks without resets or ensembles.

Problem

High update-to-data ratio off-policy RL is unstable because value functions misgeneralize to unobserved on-policy actions; prior fixes rely on resets or critic ensembles, which are costly or infeasible in real systems.

Approach

Mix a small fraction (5%) of model-generated on-policy transitions, drawn from a learned latent world model using the current policy's actions, into the replay batch for critic training, so the value function sees on-policy action outcomes and overestimation is reduced without resets or ensembles.

Key results

• At UTD 8 on dog tasks, MAD-TD matches or improves performance as update ratio rises, whereas model-free variants stagnate or deteriorate.
• MAD-TD is on par with BRO and TD-MPC2 at 2M steps with action repeat 2 and strongly outperforms TD-MPC2 and BRO at 1M steps with action repeat 2.
• Random-action model data drops MAD-TD performance close to the model-free baseline, indicating on-policy model actions are key.

Measured effects reported in the paper

Surprising findings

• Random-action model-generated data yields no benefit over the model-free baseline, showing on-policy action correction is what matters.
• Resetting barely improves MAD-TD, suggesting that resets are not needed once value misgeneralization is addressed.

Limitations

• Assumes a sufficiently strong environment model can be learned online, which may fail in novel environments.
• Model data only from states visited under past policies, leaving a state distribution shift to the target policy.
• Underperforms BRO on some simple environments, attributed to over-parameterized TD-MPC2 architecture.

Experimental setup

Datasets: DeepMind Control suite (humanoid and dog tasks) MetaWorld

Benchmarks: DeepMind Control suite hard tasks (humanoid, dog) MetaWorld

Metrics: episode return IQM mean return average regret Q-value overestimation validation and on-policy TD loss

Baselines: TD-MPC2 BRO SynthER TD3 without model data

Builds on: TD3 TD-MPC2 DYNA HL-Gauss BRO

Closest papers by meaning

• Compute-Optimal Scaling for Value-Based Deep RL (NeurIPS 2025 · similarity 0.87)
• Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates (ICLR 2024 · similarity 0.87)
• SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training (NeurIPS 2026 · similarity 0.87)
• Simplifying Deep Temporal Difference Learning (ICLR 2025 · similarity 0.87)
• The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
• The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.87)