Research map / Adversarial robustness and security / Certified and empirical robustness
Breaking the Barrier: Enhanced Utility and Robustness in Smoothed DRL Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S-DQN and S-PPO, which train DRL agents under randomized smoothing (denoiser training for DQN, median smoothing for PPO) and a Smoothed Attack for evaluating smoothed agents, achieving higher clean and robust reward with new certificates.
Problem
Smoothed (randomized smoothing) DRL agents suffer from very low clean reward and weak robustness under attack, and existing attacks overestimate their robustness.
Approach
Train agents under randomized smoothing instead of only smoothing at test time: a denoiser before the DQN Q-network trained with reconstruction plus a denoised TD loss (S-DQN), and PPO trained on median-smoothed policies (S-PPO); also a Smoothed Attack that injects the same Gaussian noise the smoothed agent uses, plus a hard-RS certified radius that does not need the Q-value range.
Key results
- S-DQN and S-PPO outperform existing smoothed agents by an average 2.16x under the strongest attack and previous robust agents by 2.13x (abstract).
- S-DQN (Vanilla) clean normalized reward 0.989 vs 0.424 for VanillaDQN+RS; under PGD at eps=0.05 it reaches 0.377 vs 0.000 for VanillaDQN+RS.
- Smoothed Attack (S-PGD) reduces S-DQN reward by 61.8% on average vs 23.6% for classic PGD, stated as 2.62x stronger.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train S-DQN with randomized smoothing and a denoiser | VanillaDQN · Atari (Pong, Freeway, RoadRunner) | clean normalized reward: 1.000 -> 0.424 | Hurt |
| Train S-DQN with a denoiser and hard randomized smoothing | VanillaDQN · Atari (Pong, Freeway, RoadRunner) | normalized reward under PGD attack (ε=0.05): 0.000 -> 0.377 | Helped |
| Use S-DQN with a RadialDQN base agent | RadialDQN · Atari (Pong, Freeway, RoadRunner) | normalized reward under PGD attack (ε=0.05): 0.210 -> 0.735 | Helped |
| Apply S-DQN with a RadialDQN base agent | RadialDQN · Atari (Pong, Freeway, RoadRunner) | clean normalized reward: 0.926 -> 0.929 | Mixed |
| Train PPO with the proposed S-PPO randomized-smoothing approach | SGLDPPO · Walker and Hopper | worst normalized reward under attacks: 0.266 -> 0.604 | Helped |
| Train PPO with the proposed S-PPO randomized-smoothing approach | WocaRPPO · Walker and Hopper | worst normalized reward under attacks: 0.342 -> 0.531 | Helped |
| Apply randomized smoothing at test time without smoothing-aware training | PPO agents · Walker and Hopper | clean normalized reward: 0.800 -> 0.740 | Hurt |
| Use Smoothed Attack (S-PGD) instead of classic PGD | S-DQN · Pong | reward: $19.4 \pm 2.1$ -> $\bf{18.4 \pm 2.1}$ | Hurt |
| Use Smoothed Attack (S-PGD) instead of classic PGD | S-DQN · Freeway | reward: $32.0 \pm 1.4$ -> $\bf{6.6 \pm 2.2}$ | Hurt |
| Use Smoothed Attack (S-PGD) instead of classic PGD | S-DQN · RoadRunner | reward: $ 17740 \pm 3718$ -> $\bf{0 \pm 0}$ | Hurt |
| Use S-PA-AD instead of PA-AD | S-DQN · Freeway | reward: $19.8 \pm 1.5$ -> $\bf{13.0 \pm 2.1}$ | Hurt |
| Use S-PA-AD instead of PA-AD | S-DQN · RoadRunner | reward: $\bf{0 \pm 0}$ -> $\bf{0 \pm 0}$ | No clear effect |
Surprising findings
- Classic PGD and PA-AD attacks are ineffective against smoothed DQN agents, giving an illusion of robustness (S-PGD reduces S-DQN reward far more).
- Previous smoothed DQN agents (e.g. VanillaDQN+RS) get zero normalized reward under PGD attack, and smoothing at test time alone does not improve PPO robustness.
Limitations
- Smoothing at test time requires Monte Carlo sampling (M samples per step), adding inference cost.
- Certified guarantees are estimated with finite samples and a confidence parameter, so they are probabilistic in practice.
- Robustness certificates are not yet used during training (stated as future work).
Experimental setup
Datasets: Atari Pong Atari Freeway Atari RoadRunner MuJoCo Walker MuJoCo Hopper
Benchmarks: Atari Pong Atari Freeway Atari RoadRunner MuJoCo Walker MuJoCo Hopper
Metrics: normalized reward clean reward reward under attack certified reward lower bound certified radius
Baselines: RadialDQN SADQN WocaRDQN CROP SGLDPPO RadialPPO WocaRPPO ATLAPPO PA-ATLAPPO Policy Smoothing
Builds on: randomized smoothing CROP Denoised Smoothing RadialDQN PA-ATLA PPO
Closest papers by meaning
- Improve Robustness of Reinforcement Learning against Observation Perturbations via l∞ Lipschitz Policy Networks (AAAI 2024 · similarity 0.88)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.88)
- Robustness Verification of Deep Reinforcement Learning Based Control Systems Using Reward Martingales (AAAI 2024 · similarity 0.87)
- Reward Certification for Policy Smoothed Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.