Research map / Reinforcement learning / Visual and representation RL
World Models as Reference Trajectories for Rapid Motor Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reflexive World Models, which freeze a pre-trained TD-MPC2 agent and train a small online residual controller to minimize latent world-model prediction error, enabling fast adaptation to actuator perturbations.
Problem
Learned control policies degrade when system dynamics change unexpectedly (e.g., actuator gain shifts), and retraining or heavy online adaptation is costly.
Approach
World model predictions are treated as implicit reference trajectories: a residual controller pi_c is trained online to make observed latent states match the forward model's predictions for the base policy, inverting the usual model-fitting gradient; the base RL policy is kept fixed.
Key results
- On Walker2D under nonstationary filtered-noise actuator perturbations, RWM reached 360.56 average reward vs 311.67 for TD-MPC2 and 233.42 for No Adaptation.
- Against TD-MPC2 pre-trained with perturbations, RWM achieved median normalized reward 0.6003 vs 0.3600 during perturbation ON.
- Online adaptation for 1M steps took about 1.8 hours on a single T4 vs 16.5 hours for full TD-MPC2 training (reported in a table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add online RWM adaptation instead of using TD-MPC2 without adaptation | TD-MPC2 · Walker2D, continuous filtered-noise perturbations | reward: 311.67 -> 360.56 | Helped |
| add online RWM adaptation instead of no adaptation | TD-MPC2 · Walker2D, continuous filtered-noise perturbations | reward: 233.42 -> 360.56 | Helped |
| add online RWM adaptation to a baseline pre-trained with actuator perturbations | TD-MPC2 · Perturbation ON, baselines pre-trained with perturbations | Median Normalized Reward: 0.3600 -> 0.6003 | Helped |
| add online RWM adaptation to a baseline pre-trained with actuator perturbations | TD-MPC2 · Perturbation ON, baselines pre-trained with perturbations | Median Normalized Control Error: 1.4162 -> 0.7307 | Helped |
| use MSE rather than a scale-invariant KL-divergence loss | RWM · motor adaptation | adaptation performance | Helped |
| add a thresholded quadratic action cost during base-policy pre-training | TD-MPC2 base policy · Humanoid action responsiveness for adaptation | action saturation and responsiveness | Helped |
Surprising findings
- Domain-randomization-pretrained TD-MPC2 outperforms RWM during nominal (perturbation OFF) dynamics, per the authors' appendix comparison.
Limitations
- Relies on assumptions (control authority, model accuracy, differentiability) that may not hold in underactuated or complex systems.
- Phased training: base policy, world model and controller are trained separately rather than end-to-end, risking controller exploitation of model errors.
- Evaluated only on actuator perturbations; morphological changes, environment shifts and sensor noise are not addressed.
Experimental setup
Datasets: DeepMind Control Suite walker-walk DeepMind Control Suite humanoid-walk 2D point-mass task
Benchmarks: DeepMind Control Suite walker-walk DeepMind Control Suite humanoid-walk 2D point-mass
Metrics: episode reward normalized median reward control error (latent prediction error) wall-clock execution time
Baselines: TD-MPC2 No Adaptation TD-MPC2 with domain randomization pre-training
Builds on: TD-MPC2 Model Reference Adaptive Control Rapid Motor Adaptation Dreamer
Closest papers by meaning
- PWM: Policy Learning with Multi-Task World Models (ICLR 2025 · similarity 0.89)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.88)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.87)
- Neural Motion Simulator Pushing the Limit of World Models in Reinforcement Learning (CVPR 2025 · similarity 0.87)
- TD-MPC2: Scalable, Robust World Models for Continuous Control (ICLR 2024 · similarity 0.87)
- EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.