Research map / Reinforcement learning / Policy gradient and actor-critic

Reward Certification for Policy Smoothed Reinforcement Learning

AAAI 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReCePS, a black-box certification method that bounds the mean cumulative reward of smoothed RL policies under various l_p perturbations by solving a convex optimisation based on f-divergences.

Problem

Certifying a provable lower bound on the total/mean cumulative reward of randomised-smoothed RL policies under adversarial observation perturbations (l1, l2 norms) and action perturbations (l0 norm), where prior methods only handled l2 observation perturbations.

Approach

Reformulates reward certification as a distributional optimisation over f-divergence balls; via a generalised Donsker-Varadhan variational formula, the worst-case expected reward becomes a convex optimisation over (nu, eta) estimated with Monte Carlo samples and Hoeffding confidence bounds, avoiding reward thresholding.

Key results

• Under l2 perturbations, the method yields tighter certified lower bounds than the CDF-based baseline, especially on Freeway, with the paper reporting Cartpole runtime 0.016s vs 0.039s per budget.
• For l1 observation perturbations, certified bounds increase with smoothing sigma, though large sigma reduces the certified reward.
• First method reported to certify l0-norm perturbations on the action space, with results on CartPole with two discrete actions.

Measured effects reported in the paper

Surprising findings

• The paper notes that as the smoothing parameter sigma grows, its bound approaches the baseline bound, attributed to higher reward variance making Monte Carlo mean estimates less accurate.

Limitations

• Monte Carlo estimation of the expectation becomes less accurate as the smoothing parameter sigma grows due to higher reward variance.
• The bound approaches the baseline for larger sigma values.
• Lipschitz-based global reward baseline is weak and only evaluated in the appendix.

Experimental setup

Datasets: CartPole Mountain Car Freeway

Benchmarks: CartPole Freeway Mountain Car

Metrics: certified lower bound of mean cumulative reward runtime per budget

Baselines: CDF-based certification (Wu et al.) Lipschitz-based global reward certification

Builds on: Policy Smoothed RL (Kumar et al.) Generalised Donsker-Varadhan variational formula DQN DDPG CDF-based reward certification

Closest papers by meaning

• Breaking the Barrier: Enhanced Utility and Robustness in Smoothed DRL Agents (ICML 2024 · similarity 0.87)
• Robustness Verification of Deep Reinforcement Learning Based Control Systems Using Reward Martingales (AAAI 2024 · similarity 0.87)
• The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards (AAAI 2025 · similarity 0.86)
• Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL (NeurIPS 2025 · similarity 0.86)
• Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption (AAAI 2024 · similarity 0.86)
• Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.86)