Research map / Reinforcement learning / Offline RL
Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MuCD, a differential-privacy-based randomized offline RL training plus certification method giving action-level and policy-level poisoning robustness guarantees for discrete and continuous, stochastic and deterministic settings.
Problem
Certifying robustness of offline RL policies against trajectory- and transition-level data poisoning, in general (continuous/stochastic) settings where prior certified methods like COPA do not apply.
Approach
Use DP-based randomized training (Sampled Gaussian Mechanism for transition-level, DP-FedAvg for trajectory-level) with policy instances, then derive per-state action certificates via inferred-score outcome guarantees and lower bounds on expected cumulative reward via expected-outcome lemma and DKW inequality.
Key results
- Policy-level certification keeps performance drop under 50% with up to 7.17% of trajectories poisoned on Freeway, versus 0.008% for COPA.
- Action-level max certified radius is about 5x larger than COPA (e.g., Freeway DQN trajectory-level max radius 37-136 vs COPA 13).
- On Half Cheetah with RDP sigma=2.0, empirical attacks leave estimated expected cumulative reward (about 61-80) well above the certified lower bound (23-49) at 10-20% poisoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase RDP training noise | DQN and C51 in Freeway and Breakout · Freeway and Breakout | action-level certified poisoning radius | Helped |
| Increase RDP training noise | DQN and C51 · Freeway and Breakout | average cumulative reward: Freeway DQN: 20.1 -> 16.0; C51: 21.3 -> 15.1; Breakout DQN: 385.4 -> 268.4; C51: 389.3 -> 102.7 | Hurt |
| Use RDP rather than ADP certification | Proposed certified defense · Atari games | policy-level robustness certification | Helped |
| Use DQN rather than C51 | Proposed certified defense · Freeway | action-level certification | Helped |
| Use DQN rather than C51 | Proposed certified defense · Breakout | action-level certification | No clear effect |
| Use proposed RDP certification rather than COPA | DQN and C51 · Freeway and Breakout, trajectory poisoning | action-level mean radius: Freeway DQN: 10.1 -> 102.4; C51: 9.7 -> 49.7; Breakout DQN: 6.6 -> 16.9; C51: 6.3 -> 21.7 | Helped |
| Use proposed defense rather than COPA | Offline RL policy · Freeway | poisoned training trajectories tolerated while certifying less than a 50% perfor: 0.008% -> 7.17% | Helped |
| Use proposed defense rather than COPA | Offline RL policy · Breakout | poisoned training trajectories tolerated while certifying less than a 50% perfor: 0.0075% -> 2.05% | Helped |
| Use transition-level rather than trajectory-level certification | Proposed policy-level certification · Offline RL | tolerable poisoning size at the same policy certification: about 10 times larger | Helped |
| Use randomized DP training with increasing noise | IQL · Half Cheetah | average cumulative reward: 96.47 -> 90.5 -> 87.0 -> 83.4 | Hurt |
Surprising findings
- DQN gives consistently higher certifications than C51 on Freeway, while matching performance on Breakout.
- Transition-level certification tolerates about 10 times more poisoned points than trajectory-level for the same policy bound, yet total transitions altered at trajectory level can be higher.
Limitations
- Uniform noise across training may limit certified radii; more adaptive noise is left for future work.
- Certified bounds are conservative relative to empirical attack performance.
- Trajectory-level and transition-level DP training are separate algorithms; no unified method yet.
Experimental setup
Datasets: D4RL Farama Gymnasium Atari Freeway Atari Breakout MuJoCo Half Cheetah
Benchmarks: Freeway Breakout Half Cheetah
Metrics: certified lower bound on expected cumulative reward action-level max and mean certified radius stability ratio average cumulative reward
Baselines: COPA DQN C51 IQL
Builds on: COPA Deep Partition Aggregation Sampled Gaussian Mechanism DP-FedAvg Rényi differential privacy
Closest papers by meaning
- Optimal Attack and Defense for Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Towards Robust Offline Reinforcement Learning under Diverse Data Corruption (ICLR 2024 · similarity 0.87)
- Data Poisoning to Fake a Nash Equilibria for Markov Games (AAAI 2024 · similarity 0.87)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Offline Safe Reinforcement Learning Using Trajectory Classification (AAAI 2025 · similarity 0.87)
- Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.