Research map / Reinforcement learning / Visual and representation RL
Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pessimistic (maximin) DQN variants with PF-RNN belief inference (BP-DQN) and DDPM-based diffusion state purification (DP-DQN) to defend RL agents against adversarial state perturbations.
Problem
RL agents at test time can be manipulated by adversarial perturbations of their state observations; existing regularization and alternating-training defenses either fail under strong attacks or are computationally expensive, and agents act on perturbed states without accounting for uncertainty about
Approach
Replace greedy action selection with a maximin choice over the epsilon-ball around the perturbed state, applied consistently in training and testing; reduce uncertainty via a belief set from a particle-filter RNN (BP-DQN) or diffusion-based purification of perturbed observations (DP-DQN), which is agnostic to the attack budget.
Key results
- On continuous Gridworld under PGD with epsilon 0.5, BP-DQN reaches 176 ± 16 reward while SA-DQN and WocaR-DQN fail (-100 and -63.2).
- On Pong under PA-AD with epsilon 15/255, DP-DQN-O achieves 19.3 ± 0.8 while DQN, SA-DQN and WocaR-DQN reach -21, -20 and -21.
- Diffusion-based DP-DQN uses one policy trained at epsilon 1/255 and remains effective across attack budgets, unlike SA-DQN and WocaR-DQN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add belief approximation to maximin search (ablation) | BP-DQN · Continuous Gridworld, PGD ε=0.5 | average episode reward: -71 \pm 91 -> 176 \pm 16 | Helped |
| add maximin search to belief approximation (ablation) | BP-DQN · Continuous Gridworld, PGD ε=0.5 | average episode reward: 45.7 \pm 134 -> 176 \pm 16 | Helped |
| add maximin search to DDPM-based belief (ablation) | DP-DQN-O · Pong, PA-AD ε=15/255 | average episode reward: 18.8 \pm 1.6 -> 19.3 \pm 0.8 | Helped |
| use belief-enriched pessimistic Q-learning instead of the strongest listed baseline | Continuous Gridworld · Continuous Gridworld, PGD ε=0.5 | average episode reward: -63.2 \pm 88 -> 176 \pm 16 | Helped |
| use diffusion-based state purification instead of WocaR-DQN | DP-DQN-O · Pong, PA-AD ε=15/255 | average episode reward: -21 \pm 0 -> 19.3 \pm 0.8 | Helped |
| use diffusion-based state purification instead of WocaR-DQN | DP-DQN-O · Freeway, PA-AD ε=15/255 | average episode reward: 21.6 \pm 1 -> 28.8 \pm 1.3 | Helped |
Surprising findings
- Under no attack or weak attacks, the proposed methods lose some reward relative to SA-DQN and WocaR-DQN on Atari.
- The paper argues that perturbed Atari pixel states are mostly invalid states, which motivates diffusion-based purification.
Limitations
- Requires access to a clean environment during training.
- Diffusion-based purification adds substantial test-time computational overhead.
- Only value-based methods are studied; policy-gradient extensions are left to future work.
Experimental setup
Datasets: Continuous Gridworld Atari Pong Atari Freeway
Benchmarks: Continuous Gridworld Atari Pong Atari Freeway
Metrics: average episode reward
Baselines: DQN SA-DQN WocaR-DQN Radial-DQN
Builds on: DQN SA-MDP WocaR-RL PF-RNN DDPM Progressive Distillation
Closest papers by meaning
- Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.88)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Diffusion Guided Adversarial State Perturbations in Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula (ICLR 2024 · similarity 0.87)
- RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations (ICML 2025 · similarity 0.87)
- Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.