Research map / Reinforcement learning / Visual and representation RL
Neural Motion Simulator Pushing the Limit of World Models in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MoSim, a neural ODE world model with rigid-body inductive biases and residual correctors that predicts robot motion states long-horizon, and uses it to enable zero-shot model-based RL with TQC/SAC.
Problem
Existing world models are mostly evaluated indirectly through downstream RL tasks, leaving their direct multi-step physical state prediction accuracy unmeasured; long-horizon accurate motion prediction is needed for zero-shot policy learning in imagined worlds.
Approach
Model robot dynamics as a rigid-body predictor (Cholesky-parameterized inertia matrix, state and action encoders) plus a residual corrector, integrated with a Neural ODE (DOPRI5) and trained in multiple stages (predictor first, then frozen-predictor corrector); the simulator predicts raw states so any model-free RL algorithm can be wrapped into a model-based one.
Key results
- MoSim achieves much lower multi-step MSE than DreamerV3 on Cheetah, Reacher, Acrobot, Panda, Go2 and Humanoid (e.g., Reacher 100-step: 0.0009 vs 0.0988 DreamerV3-r).
- On TD-MPC2 latent-space evaluation, MoSim reduces 3-step prediction MSE relative to TD-MPC2 (e.g., Reacher 4.8e-5 vs 2.9e-7).
- Zero-shot RL with MoSim achieves scores on three DM Control tasks that closely approximate real-environment scores; cheetah-run plateaus around 100 under zero-shot training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use random-policy data rather than Dreamer replay-buffer experience data for training | MoSim · OOD TD-MPC2 policy data, Reacher | MSE: 0.0027 -> 0.0005 | Helped |
| Add the rigid-body dynamics inductive bias to the predictor (ablation) | MoSim predictor · Hopper-hop | prediction performance | Helped |
| Use multistage predictor-then-corrector training instead of end-to-end training (ablation) | MoSim · Hard tasks, including Hopper-hop | prediction performance | Helped |
| Use multistage predictor-then-corrector training instead of end-to-end training (ablation) | MoSim · Hopper-hop | training efficiency | Helped |
| Train with observation noise (ablation) | MoSim · Reacher, 100 steps | MSE: 0.0009 -> 0.0008 | Helped |
| Train with observation noise (ablation) | MoSim · Panda, 100 steps | MSE: 0.0043 -> 0.0042 | Helped |
| Train with observation noise (ablation) | MoSim · Go2, 100 steps | MSE: 0.1282 -> 0.2111 | Hurt |
| Reduce the policy-training prediction horizon (ablation) | MoSim-based RL · Policy learning | final policy performance | Hurt |
| Fit a residual flow and penalize low-density states in the reward (ablation) | Zero-shot MoSim-based Cheetah RL · Cheetah Run | reward | Helped |
| Initialize the virtual environment from replay-buffer states and periodically collect real data | MoSim-based few-shot RL · Cheetah Run | score | No clear effect |
Surprising findings
- Models trained on random-action data generalized better than those trained on replay-buffer experience data on out-of-distribution TD-MPC2 trajectories.
- Adding Gaussian observation noise to training data did not degrade MoSim's performance on several tasks.
Limitations
- Zero-shot learning fails on cheetah-run because the model cannot maintain accurate predictions beyond about 100 steps and suffers distribution shift during training.
- Few-shot variant still plateaus at a relatively low score on cheetah, and a notable gap remains for zero-shot performance on many tasks.
- Assumes deterministic dynamics; stochasticity handled only via ensemble sampling and noise robustness tests.
Experimental setup
Datasets: DeepMind Control Suite random Poisson-action rollouts TD-MPC2 policy rollouts Panda Go2
Benchmarks: DeepMind Control Suite (Cheetah, Reacher, Acrobot, Hopper, Humanoid) Panda Go2
Metrics: MSE prediction loss Lyapunov characteristic exponent episode return
Baselines: DreamerV3 TD-MPC2 RSSM
Builds on: Neural ODE DreamerV3 TD-MPC2 Recurrent State-Space Model
Closest papers by meaning
- DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (ICML 2025 · similarity 0.88)
- Learning 3D Persistent Embodied World Models (NeurIPS 2025 · similarity 0.87)
- WALL-E: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents (NeurIPS 2025 · similarity 0.87)
- DyMoDreamer: World Modeling with Dynamic Modulation (NeurIPS 2025 · similarity 0.87)
- World Models as Reference Trajectories for Rapid Motor Adaptation (NeurIPS 2025 · similarity 0.87)
- From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.