Research map / Reinforcement learning / Policy gradient and actor-critic
Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L1-MBRL, an add-on that affinizes a learned nonlinear dynamics model via a switching law and applies L1 adaptive control to the MBRL input, improving METRPO and MBMF on MuJoCo tasks with and without noise.
Problem
Model-based RL policies are vulnerable to model (epistemic) and aleatoric uncertainties, causing sim-to-real gaps and poor robustness; classical robust/adaptive control assumes control-affine or parametric models that do not match nonlinear learned NN dynamics.
Approach
Approximate the learned nonlinear transition model by a sequence of control-affine models (first-order Taylor expansion in the input, re-linearized when the affinization error exceeds a threshold), then add an L1 adaptive control input computed from this affine model to the MBRL control, while training on the nominal RL input.
Key results
- L1-METRPO outperforms METRPO in most noise-free and noisy settings, e.g., Hopper noise-free 1491.4 vs 1140.1 and Halfcheetah noise-free 2588.6 vs 2367.3.
- With L1 augmentation, MB-MF Halfcheetah noise-free rises from 126.9 to 304.5 and Inverted Pendulum from -100.5 to -10.5.
- Training on noise-free environment and testing with L1 under action noise improves METRPO's Hopper result from 198.9 to 623.4 (sim-to-real ablation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add L1 adaptive control to METRPO | METRPO · MuJoCo environments, noise-free and noisy | average reward | Mixed |
| Add L1 adaptive control to MBMF | MBMF · MuJoCo environments, noise-free and noisy | average reward | Helped |
| Apply L1 control during testing, not training (ablation) | METRPO · MuJoCo environments under noise | average reward | Helped |
| Apply L1 control during training, not testing (ablation) | METRPO · MuJoCo environments | average reward | Mixed |
| Apply L1 control during both training and testing (ablation) | METRPO · MuJoCo environments | average reward | Helped |
| Add L1 control at deployment to a policy trained without L1 (ablation) | METRPO · Inverted Pendulum, action noise σa=0.1 | average reward: 30.2± 45.1 -> -0.0 ± 0.0 | Helped |
| Add L1 control at deployment to a policy trained without L1 (ablation) | METRPO · Swimmer, observation noise σo=0.1 | average reward: 337.8 ± 2.9 -> 331.2 ± 8.34 | Hurt |
| Add L1 control at deployment to a policy trained without L1 (ablation) | METRPO · Walker, action noise σa=0.1 | average reward: -6.0 ± 0.8 -> -6.3 ± 0.7 | Hurt |
| Use the control-affine model instead of a fully nonlinear model (ablation) | METRPO · Halfcheetah | average reward | Hurt |
Surprising findings
- Control-affine learned models alone fail to learn Halfcheetah dynamics, motivating the switching-law affinization instead.
- L1 augmentation applied during training as well as testing gives the best results, indicating it shifts the training data distribution beneficially.
Limitations
- Performance is tied to the baseline MBRL algorithm; L1 cannot independently guarantee good performance.
- Choice of affinization threshold epsilon_a trades off robustness against baseline fidelity and requires heuristic tuning.
- Theoretical guarantees rely on a known bound on the learned-model error that cannot be guaranteed in the real environment.
Experimental setup
Datasets: OpenAI Gym MuJoCo environments (Inverted Pendulum, Swimmer, Hopper, Walker, Halfcheetah)
Benchmarks: Inverted Pendulum Swimmer Hopper Walker Halfcheetah
Metrics: average episode reward (mean and std over seeds, final windo
Baselines: METRPO MBMF
Builds on: METRPO MBMF L1 adaptive control guided policy search TRPO
Closest papers by meaning
- Reinforcement Learning under Model Mismatch (NeurIPS 2017 · similarity 0.89)
- Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Robust Inverse Constrained Reinforcement Learning under Model Misspecification (ICML 2024 · similarity 0.87)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- SOMBRL: Scalable and Optimistic Model-Based RL (NeurIPS 2025 · similarity 0.87)
- Model-Free Robust $\phi$-Divergence Reinforcement Learning Using Both Offline and Online Data (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.