Research map / Reinforcement learning / Visual and representation RL
Improve Robustness of Reinforcement Learning against Observation Perturbations via l∞ Lipschitz Policy Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SortRL, a DRL policy network built from an l_inf 1-Lipschitz SortNet whose output margin certifies a robust radius, trained by policy distillation from a PPO/DQN teacher, to resist l_inf observation perturbations.
Problem
DRL policies are vulnerable to small l_inf-bounded perturbations of observations; existing defenses rely on regularizers or adversarial training that hurt nominal performance or add costly attackers.
Approach
Build the policy from an l_inf 1-Lipschitz SortNet so that the action-score margin gives a provable lower bound on the l_inf robust radius (radius >= margin/2), and train the student by distillation with a hinge robustness loss that pushes margins above 2*epsilon.
Key results
- On Atari RoadRunner at epsilon=5/255, SortRL-DQN reaches 40905 episode reward vs 31545 for RADIAL-A3C (about 29.6% higher).
- Relative to nominal performance, SortRL loses under 1.7% at epsilon=5/255 while RADIAL loses about 18%.
- ACR for SortRL exceeds 99.6% across tested Atari tasks at 1/255.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SortRL-DQN instead of RADIAL-DQN | DQN policies · Freeway, ε=5/255 | episode reward: 29.1\pm0.17 -> 33.92\pm0.33 | Helped |
| Use SortRL-DQN instead of RADIAL-A3C | RoadRunner policies · RoadRunner, ε=5/255 | episode reward: 31545\pm1480 -> 40905\pm1249 | Helped |
| Use SortRL-DQN instead of RADIAL-DQN | Pong policies · Pong, ε=5/255 | episode reward: 21.0\pm0.0 -> 21.0\pm0.0 | No clear effect |
| Use SortRL-RADIAL instead of RADIAL-DQN | BankHeist policies · BankHeist, ε=5/255 | episode reward: 1182.5\pm43.3 -> 1341.1\pm5.1 | Helped |
| Use SortRL-PPO instead of RADIAL-PPO | PPO policies · Jumper test, ε=5/255 | episode reward: 3.59\pm0.15 -> 4.65\pm0.39 | Helped |
| Use SortRL-PPO instead of RADIAL-PPO | PPO policies · Coinrun test, ε=5/255 | episode reward: 6.67\pm0.15 -> 7.20\pm0.41 | Helped |
| Use SortRL instead of BCL-RADIAL | Freeway policies · Freeway, ε=20/255 | episode reward: 21.2 -> 27.2 | Helped |
| Apply SortRL robustness training | Atari policies · Atari video games, ε=5/255 | episode reward relative to nominal performance: less than 1.7% | Helped |
| Use SortRL instead of standard PPO | Classic control policies · Classic control, ε>0.1 | episode reward | Helped |
| Use SortRL instead of standard PPO | MountainCar and LunarLander policies · Classic control, ε=0.0 | episode reward | Hurt |
Surprising findings
- Some methods (e.g., RADIAL and SortRL on Coinrun) score higher under perturbation than in the nominal environment.
- SortRL outperforms standard DRL in some nominal tasks such as Freeway, Jumper, and Coinrun.
Limitations
- Nominal performance drop on some classic control tasks such as MountainCar and LunarLander at epsilon=0.
- Requires a pretrained teacher policy and distillation, adding a training stage.
- Robust radius guarantee is a lower bound based on the margin, so it is conservative.
Experimental setup
Datasets: CartPole Acrobot MountainCar LunarLander Atari (Freeway, RoadRunner, Pong, BankHeist) ProcGen (Jumper, Coinrun)
Benchmarks: CartPole Acrobot MountainCar LunarLander Atari Freeway Atari RoadRunner Atari Pong Atari BankHeist ProcGen Jumper ProcGen Coinrun
Metrics: episode reward under l_inf PGD perturbations action certification rate (ACR) normalized score
Baselines: PPO DQN A3C RS-DQN SA-DQN WocaR-DQN RADIAL-DQN RADIAL-A3C BCL-RADIAL BCL-MOS-AT
Builds on: SortNet Policy Distillation SA-MDP PPO DQN
Closest papers by meaning
- Breaking the Barrier: Enhanced Utility and Robustness in Smoothed DRL Agents (ICML 2024 · similarity 0.88)
- Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach (NeurIPS 2024 · similarity 0.88)
- RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations (ICML 2025 · similarity 0.88)
- LipsNet++: Unifying Filter and Controller into a Policy Network (ICML 2025 · similarity 0.88)
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.