Research map / Reinforcement learning / Visual and representation RL
Differentiable Information Enhanced Model-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MB-MIX, a model-based RL method for differentiable environments that combines trajectory-length mixing (to reduce policy-gradient variance) with Sobolev training of dynamics models using first-order gradient matching, and reports gains on DiffRL, Brax, DaXBench and a bipedal robot.
Problem
Policy training in differentiable environments is unstable due to collisions and exploding/vanishing gradients over long trajectories, and learned dynamics models accumulate errors in both state and gradient predictions.
Approach
Mix policy-gradient objectives over trajectories of different lengths (a TD(lambda)-style weighted average of path-derivative returns) to reduce variance, and train the dynamics model with a Sobolev loss that also matches Jacobians of the transition function so that model training is consistent with path-derivative policy training.
Key results
- On DiffRL, MB-MIX achieves the highest reward on all six tasks, e.g., Ant 6363 vs. SHAC 5174 and MAAC 5289, and Humanoid 4955 vs. SHAC 3321.
- On DaXBench, MIX reaches 0.73 on Unfold-Cloth-1 versus 0.50 for SHAC and 0.42 for APG.
- The paper claims MIX variance is bounded below SHAC's for any trajectory length when lambda and gamma are below 1 (Theorem 1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MB-MIX instead of MAAC | DiffRL Ant · DiffRL Ant | reward: 5289\err{651} -> \textbf{6363}\err{27} | Helped |
| Use MB-MIX instead of SHAC | DiffRL Cheetah · DiffRL Cheetah | reward: \textbf{8366}\err{259} -> \textbf{8483}\err{80} | Helped |
| Use MB-MIX instead of SHAC | DiffRL Hopper · DiffRL Hopper | reward: 3638\err{222} -> \textbf{4048}\err{127} | Helped |
| Use MB-MIX instead of SHAC | DiffRL Cartpole · DiffRL Cartpole | reward: -622\err{25} -> \textbf{-604}\err{9} | Helped |
| Use MB-MIX instead of SHAC | DiffRL Humanoid · DiffRL Humanoid | reward: 3321\err{968} -> \textbf{4955}\err{197} | Helped |
| Use MB-MIX instead of SHAC | DiffRL SNU-Humanoid · DiffRL SNU-Humanoid | reward: 3722\err{395} -> \textbf{3907}\err{94} | Helped |
| Use MIX instead of APG | DaXBench · DaXBench Fold-Cloth-1 | performance: 0.36\err{0.06} -> \textbf{0.52\err{0.02}} | Helped |
| Use MIX instead of SHAC | DaXBench · DaXBench Fold-Cloth-3 | performance: \textbf{0.22 \err{0.22}} -> \textbf{0.23\err{0.08}} | Helped |
| Use MIX instead of SHAC | DaXBench · DaXBench Unfold-Cloth-1 | performance: 0.50 \err{0.03} -> \textbf{0.73\err{0.01}} | Helped |
| Use MIX instead of SHAC | DaXBench · DaXBench Unfold-Cloth-3 | performance: \textbf{0.48 \err{0.03}} -> \textbf{0.51\err{0.03}} | Helped |
| Add trajectory length mixing (ablation) | Gradient-based policy optimization · Tabular random MDP | average reward | Helped |
| Add trajectory length mixing (ablation) | Gradient-based policy training · DiffRL Ant and Cheetah | reward | Helped |
Limitations
- Theoretical analysis covers only the model-free MIX estimator, not the model-based setting.
- Gains reported on DaXBench and Brax are modest or confined to few tasks; Brax results are only shown in figures.
Experimental setup
Datasets: DiffRL Brax DaXBench
Benchmarks: DiffRL Brax DaXBench Bruce Fast Run
Metrics: episode reward training stability (variance across seeds)
Baselines: SHAC PPO SAC DreamerV3 LOOP MAAC APG MIX
Builds on: SHAC Sobolev training MBPO SVG DiffRL
Closest papers by meaning
- Gradient Information Matters in Policy Optimization by Back-propagating through Model (ICLR 2022 · similarity 0.89)
- Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation (ICML 2024 · similarity 0.88)
- PODS: Policy Optimization via Differentiable Simulation (ICML 2021 · similarity 0.88)
- Differentiable MPC for End-to-end Planning and Control (NeurIPS 2018 · similarity 0.88)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.87)
- Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.