Research map / Adversarial robustness and security / Certified and empirical robustness

Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach

NeurIPS 2024 · Adversarial robustness and security › Certified and empirical robustness · approach: Adversarial training with feature consistency · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows deep RL policies (SAC, TD3, Dreamer V3) are chaotic under small state perturbations in DeepMind Control tasks, and proposes adding a variance-based Maximal Lyapunov Exponent regulariser to Dreamer V3's policy loss to improve robustness to observation noise.

Problem

Deep RL policies for continuous control are sensitive to small state perturbations and observation noise; their closed-loop dynamics can be chaotic, making rewards and trajectories unpredictable and fragile in real-world use.

Approach

Estimate the Maximal Lyapunov Exponent of the learned policy via variance across stochastic RSSM rollouts in Dreamer V3, and add this variance term as a regulariser to the policy loss so the agent favours stable state trajectories while maximising return.

Key results

• MLE regularisation reduced Dreamer V3's MLE in most environments, e.g. Walker Stand from 0.1688 to 0.0654 and Pointmass from 0.0326 to -0.0275, while reward stayed similar in most tasks.
• Under Gaussian observation noise, MLE DR3 significantly improved performance in four of six noisy control systems, while Walker Stand and Cheetah Run were similar to baseline.
• All deep RL policies (SAC, TD3, Dreamer V3) showed positive MLE in high-dimensional Walker and Cheetah tasks, indicating chaotic state dynamics.

Measured effects reported in the paper

Surprising findings

• Cartpole Balance and Cartpole Swingup policies learned by SAC and TD3 were stable, but Dreamer V3 produced low-level chaos even in these simple environments.
• Dreamer V3 with MLE regularisation still had positive MLE in most environments, so the trajectories remained chaotic though less divergent.

Limitations

• Regularisation assumes RSSM accurately estimates local state divergence; may be less effective where the RSSM struggles to model dynamics.
• MLE remains positive in most environments after regularisation, so trajectories are still chaotic.
• Only L=3 samples used for variance estimation due to compute constraints.

Experimental setup

Datasets: DeepMind Control Suite Pointmass Cartpole Balance Cartpole Swingup Walker Stand Walker Walk Walker Run Cheetah Run

Benchmarks: DeepMind Control Suite

Metrics: episode reward maximal Lyapunov exponent sum of Lyapunov exponents reward MLE interquartile mean reward under Gaussian observation noise

Baselines: SAC TD3 Dreamer V3

Builds on: Dreamer V3 Soft Actor-Critic Benettin et al. Lyapunov spectrum estimation Stable Baselines3

Closest papers by meaning

• Robustness Verification of Deep Reinforcement Learning Based Control Systems Using Reward Martingales (AAAI 2024 · similarity 0.89)
• Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Improve Robustness of Reinforcement Learning against Observation Perturbations via l∞ Lipschitz Policy Networks (AAAI 2024 · similarity 0.88)
• Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula (ICLR 2024 · similarity 0.88)
• State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.88)