Research map / Reinforcement learning / Policy gradient and actor-critic
Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Certifies stability of RL policies by augmenting learned value functions with neural residuals into multi-step (generalized) Lyapunov functions, and extends this to joint training of neural controllers and certificates with larger certified regions of attraction.
Problem
Classical Lyapunov certificates require strict stepwise decrease, which is hard to construct for learned RL policies; RL value functions do not naturally satisfy Lyapunov conditions, especially under discounting.
Approach
Relax the Lyapunov decrease to a weighted average decrease over M steps, and form the certificate as the RL value function plus a learned neural residual with learned state-dependent step weights; for LQR this yields LMI certificates valid for a wider range of discount factors.
Key results
- Generalized decrease condition F(x)<=0 holds for 100% of sampled test states for PPO, SAC and TD-MPC policies on inverted pendulum and cartpole.
- On inverted pendulum joint synthesis, certified ROA volume rises from 42.9 (M=1) to 76.7 (M=2) and 89.2 (M=3).
- For the scalar LQR example, one-step LMIs certify gamma > 0.8090 while optimized M=2 weights certify gamma > 0.6229 (true threshold 1/3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M = 2 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · Inverted Pendulum | certified ROA volume: 42.9 $\pm$ 1.2 -> 76.7 $\pm$ 1.3 | Helped |
| Use M = 3 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · Inverted Pendulum | certified ROA volume: 42.9 $\pm$ 1.2 -> 89.2 $\pm$ 1.2 | Helped |
| Use M = 2 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · Path Tracking | certified ROA volume: 21.8 $\pm$ 0.6 -> 23.6 $\pm$ 0.5 | Helped |
| Use M = 3 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · Path Tracking | certified ROA volume: 21.8 $\pm$ 0.6 -> 23.9 $\pm$ 0.5 | Helped |
| Use M = 2 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · 2D Quadrotor | certified ROA volume: 103.5 $\pm$ 1.8 -> 109.1 $\pm$ 2.0 | Helped |
| Use M = 3 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · 2D Quadrotor | certified ROA volume: 103.5 $\pm$ 1.8 -> 113.7 $\pm$ 2.0 | Helped |
| Use M = 2 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · Inverted Pendulum | verification time (seconds): 11.7 $\pm$ 0.5 -> 21.5 $\pm$ 0.8 | Hurt |
| Use M = 3 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · Inverted Pendulum | verification time (seconds): 11.7 $\pm$ 0.5 -> 39.2 $\pm$ 1.3 | Hurt |
| Use M = 2 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · Path Tracking | verification time (seconds): 8.6 $\pm$ 0.5 -> 19.5 $\pm$ 1.0 | Hurt |
| Use M = 3 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · Path Tracking | verification time (seconds): 8.6 $\pm$ 0.5 -> 36.7 $\pm$ 1.3 | Hurt |
| Use M = 2 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · 2D Quadrotor | verification time (seconds): 2209.5 $\pm$ 72.6 -> 3858.7 $\pm$ 112.4 | Hurt |
| Use M = 3 instead of M = 1 (ablation) | Generalized Lyapunov training and verification · 2D Quadrotor | verification time (seconds): 2209.5 $\pm$ 72.6 -> 5628.6 $\pm$ 185.9 | Hurt |
Surprising findings
- Learned step weights concentrate on the final steps of the horizon (e.g., 36-38% in the last 20% of steps), tolerating early non-monotonic transients.
- Multi-step verification is much slower than one-step (e.g., cartpole-scale quadrotor verification 2209 s for M=1 vs 5629 s for M=3).
Limitations
- Certification horizon M is fixed and minimal viable M is unclear.
- Not yet applied to high-dimensional systems such as humanoids or dexterous manipulators.
- Formal verification with learned state-dependent weights and certificates remains computationally challenging.
Experimental setup
Datasets: Gymnasium inverted pendulum Gymnasium cartpole DeepMind Control Suite
Benchmarks: Gymnasium inverted pendulum Gymnasium cartpole inverted pendulum (verification) path tracking (verification) 2D quadrotor (verification)
Metrics: percentage of test states satisfying generalized decrease co certified ROA volume verification time
Baselines: classical one-step Lyapunov (M=1) PPO SAC TD-MPC one-step Lyapunov LMIs
Builds on: Lyapunov stability theory discounted LQR Lyapunov LMIs k-induction Lyapunov-guided neural controller-certificate training (Chang et al.)
Closest papers by meaning
- Analytical Lyapunov Function Discovery: An RL-based Generative Approach (ICML 2025 · similarity 0.89)
- Lyapunov-stable Neural Control for State and Output Feedback: A Novel Formulation (ICML 2024 · similarity 0.88)
- Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach (NeurIPS 2024 · similarity 0.87)
- Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.86)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.86)
- Two‑Stage Learning of Stabilizing Neural Controllers via Zubov Sampling and Iterative Domain Expansion (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.