Research map / Reinforcement learning / Visual and representation RL
Residual-MPPI: Online Policy Customization for Continuous Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Residual-MPPI, an online sampling-based planner that customizes a fixed maximum-entropy prior policy to new add-on rewards by scoring MPPI samples with the prior's log-likelihood plus an add-on reward, evaluated in MuJoCo and Gran Turismo Sport with GT Sophy 1.0.
Problem
Adapting a trained continuous-control policy to new add-on requirements (e.g., safety, behavior preferences) without retraining, extra data, or access to the original reward or training parameters.
Approach
Integrates residual Q-learning's augmented objective (add-on reward plus omega' times prior log-likelihood) into MPPI, so the planner samples action sequences from the prior and optimizes them online using a learned dynamics model; no policy training is needed at execution time.
Key results
- In GTS, zero-shot Residual-MPPI reduces off-course steps from 93.13 (GT Sophy 1.0) to 9.03 with lap time 123.34 s; few-shot reaches 4.43 steps and 122.93 s.
- In MuJoCo HalfCheetah, zero-shot Residual-MPPI reaches total reward 1936.2 versus 1000.7 for the prior policy, with a hind-leg angle of 0.02 versus 0.14.
- Residual-SAC needed about 80,000 laps to reach its GTS performance, while Residual-MPPI used about 2,000 laps for dynamics training plus about 100 online laps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| customize the prior policy with Residual-MPPI rather than use the prior policy unchanged | SAC prior policy · HalfCheetah | total reward: 1000.7 \pm 88.8 -> 1936.2 \pm 109.3 | Helped |
| include the prior-policy log-likelihood term rather than use Greedy-MPPI | MPPI customization of SAC prior policy · Ant | total reward: 6104.2 \pm 1532.0 -> 6846.7 \pm 647.8 | Helped |
| include the prior-policy log-likelihood term rather than use Greedy-MPPI | MPPI customization of GT Sophy 1.0 · GTS | lap completion: *Failed -> 123.34 \pm 0.22 | Helped |
| include prior-policy action sampling rather than use Full-MPPI | MPPI customization of GT Sophy 1.0 · GTS | lap completion: *Failed -> 123.34 \pm 0.22 | Helped |
| fine-tune the dynamics model with online planner data (ablation) | Zero-shot Residual-MPPI · GTS | lap time: 123.34 \pm 0.22 -> 122.93 \pm 0.14 | Helped |
| fine-tune the dynamics model with online planner data (ablation) | Zero-shot Residual-MPPI · GTS | off-course steps: 9.03 \pm 3.33 -> 4.43 \pm 2.39 | Helped |
| customize GT Sophy 1.0 to reduce off-course driving | GT Sophy 1.0 · GTS | off-course steps: 93.13 \pm 1.98 -> 9.03 \pm 3.33 | Helped |
| customize GT Sophy 1.0 to stay on course | GT Sophy 1.0 · GTS | lap time: 117.77 \pm 0.08 -> 123.34 \pm 0.22 | Hurt |
| train Residual-SAC with 2K laps instead of online planning | GT Sophy 1.0 customization · GTS | lap completion: *Failed | Hurt |
Surprising findings
- Full-MPPI, which has full reward access but no prior sampling, fails on most MuJoCo tasks (e.g., Hopper total reward 20.5), and Guided-MPPI and Greedy-MPPI fail to complete any GTS lap.
- Residual-SAC at 2K laps could not finish a single lap in GTS, showing RL is far less data-efficient than the planner here.
Limitations
- Performance depends on an accurate learned dynamics model; sim-to-real is not demonstrated.
- Finite planning horizon requires a long horizon because no terminal value estimator is used.
- Customized GTS policies still have some off-course steps (9 zero-shot, 4.4 few-shot).
Experimental setup
Datasets: MuJoCo rollouts from SAC prior policies GTS driving rollouts from GT Sophy 1.0
Benchmarks: HalfCheetah Swimmer Hopper Ant Gran Turismo Sport
Metrics: total reward basic reward add-on reward lap time off-course steps
Baselines: Prior Policy Greedy-MPPI Full-MPPI Guided-MPPI Valued-MPPI Residual-SAC Fulltask-SAC
Builds on: Residual Q-learning Model Predictive Path Integral Soft Actor-Critic Residual-MCTS
Closest papers by meaning
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.86)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.85)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.84)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.84)
- TD-MPC2: Scalable, Robust World Models for Continuous Control (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.