Research map / Reinforcement learning / Policy gradient and actor-critic

Robustness Verification of Deep Reinforcement Learning Based Control Systems Using Reward Martingales

AAAI 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces reward martingales (upper/lower reward supermartingales and submartingales) to certify expected cumulative reward bounds and tail bounds for DRL-based control systems under state perturbations, trained as neural networks.

Problem

Robustness verification of DRL-based control systems under state perturbations: certifying bounds on expected cumulative rewards and tail probabilities of cumulative rewards.

Approach

Define upper reward supermartingales and lower reward submartingales (with pre-expectation conditions over perturbed dynamics) that bound expected cumulative reward via the Optional Stopping Theorem and tail probabilities via Hoeffding's inequality; train these as neural networks in a train-validate loop, using either over-approximation (for continuous policies) or an analytic computation (for abs

Key results

• Certified upper and lower bounds on expected cumulative reward tightly enclose simulation outcomes (mean and std over 200 episodes per initial state) across CartPole, MountainCar, B1 and B2.
• Certified tail bounds on cumulative reward tightly enclose statistical tail probabilities for CartPole and B1 under Gaussian and uniform perturbations.
• The analytic approach is up to 35.32% faster in training and validating than the over-approximation approach.

Measured effects reported in the paper

Surprising findings

• Tightness of certified bounds does not show a consistent trend with noise magnitude, even though system performance decreases with noise.

Limitations

• Assumes deterministic dynamics and policies; probabilistic ones left as future work.
• Over-approximation approach may produce false positives and timeouts in high dimensions.
• Bound tightness depends on system, policy and perturbation and is not uniformly predictable.

Experimental setup

Datasets: CartPole (Gym) MountainCar (Gym) B1 (reachability benchmark) B2 (reachability benchmark)

Benchmarks: CartPole MountainCar B1 B2

Metrics: certified upper/lower bounds on expected cumulative reward tail probability bounds training and validation time

Builds on: ranking supermartingales Optional Stopping Theorem Hoeffding's inequality on martingales interval bound propagation

Closest papers by meaning

• Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach (NeurIPS 2024 · similarity 0.89)
• Breaking the Barrier: Enhanced Utility and Robustness in Smoothed DRL Agents (ICML 2024 · similarity 0.87)
• Reward Certification for Policy Smoothed Reinforcement Learning (AAAI 2024 · similarity 0.87)
• Improve Robustness of Reinforcement Learning against Observation Perturbations via l∞ Lipschitz Policy Networks (AAAI 2024 · similarity 0.87)
• Time-Constrained Robust MDPs (NeurIPS 2024 · similarity 0.86)
• Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach (ICML 2024 · similarity 0.86)