Research map / Reinforcement learning / Policy gradient and actor-critic
Bisimulation Metric for Model Predictive Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BS-MPC, which adds a per-timestep on-policy bisimulation metric loss to TD-MPC's objective to train the encoder directly, improving stability, noise robustness and training speed on DM Control.
Problem
Model-based RL with latent planning (TD-MPC) suffers training instability/collapse, vulnerability to distractor noise in image inputs, and high computation time because the encoder is only indirectly trained.
Approach
Add an explicit pi*-bisimulation metric loss (MSE between l1 latent distance and on-policy bisimulation distance with latent W2 dynamics term) at every time step to train the encoder, and make the latent computation parallel across time steps by encoding each observation directly rather than rolling out latent dynamics sequentially.
Key results
- On state-based DMControl, BS-MPC avoids the late-training TD-MPC collapse and outperforms TD-MPC, SAC and Dreamer-v3 on average, with the largest gaps on dog and humanoid tasks.
- On 5 image tasks with Kinetics distractors, BS-MPC constantly outperforms TD-MPC in every environment.
- Training time is reduced versus TD-MPC, e.g. 2.0h vs 2.4h on Cartpole-Swingup and 8.3h vs 10.0h on Finger-Spin (same hardware RTX-4090).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add explicit bisimulation metric loss and parallelize objective computation, compared with TD-MPC | BS-MPC · Cartpole-Swingup | training time (hours): 2.4 -> 2.0 | Helped |
| Add explicit bisimulation metric loss and parallelize objective computation, compared with TD-MPC | BS-MPC · Cheetah-run | training time (hours): 4.8 -> 4.0 | Helped |
| Add explicit bisimulation metric loss and parallelize objective computation, compared with TD-MPC | BS-MPC · Finger-Spin | training time (hours): 10.0 -> 8.3 | Helped |
| Add explicit bisimulation metric loss and parallelize objective computation, compared with TD-MPC | BS-MPC · Walker-run | training time (hours): 10.1 -> 8.3 | Helped |
| Add an explicit bisimulation metric loss to the encoder objective | BS-MPC · 26 DM Control state-based tasks | episode return | Helped |
| Add an explicit bisimulation metric loss to the encoder objective | BS-MPC · 10 DM Control image-based tasks | episode return | No clear effect |
| Add an explicit bisimulation metric loss to the encoder objective | BS-MPC · 5 DM Control image-based tasks with Kinetics distractions | episode return | Helped |
Surprising findings
- On image-based tasks BS-MPC is only competitive with TD-MPC, DrQ-v2 and Dreamer-v3, and TD-MPC2 converges faster on some tasks (quadruped-run, walker-run).
Limitations
- Requires extensive per-environment tuning of the bisimulation loss weight c4, selected by grid search.
- Theoretical value bound assumes access to true dynamics and reward and uses a learned-policy convergence assumption.
Experimental setup
Datasets: DeepMind Control Suite Kinetics driving videos (distractor backgrounds)
Benchmarks: DeepMind Control Suite (26 state-based tasks) DeepMind Control Suite (10 image-based tasks) DMControl image tasks with Kinetics distractors (5 tasks)
Metrics: episode return training wall-clock time training stability
Baselines: TD-MPC SAC Dreamer-v3 DrQ-v2 CURL TD-MPC2
Builds on: TD-MPC on-policy bisimulation metric pi*-bisimulation metric MPPI TD-MPC2
Closest papers by meaning
- A Generalized Bisimulation Metric of State Similarity between Markov Decision Processes: From Theoretical Propositions to Applications (NeurIPS 2025 · similarity 0.86)
- Bootstrapped Model Predictive Control (ICLR 2025 · similarity 0.85)
- Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control (NeurIPS 2024 · similarity 0.84)
- State Chrono Representation for Enhancing Generalization in Reinforcement Learning (NeurIPS 2024 · similarity 0.83)
- Differentiable Information Enhanced Model-Based Reinforcement Learning (AAAI 2025 · similarity 0.83)
- Discrete Codebook World Models for Continuous Control (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.