Research map / Reinforcement learning / Policy gradient and actor-critic
Confounding Robust Deep Reinforcement Learning: A Causal Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Causal-DQN, a DQN variant that learns lower-bound Q-values from confounded demonstrator data via a causal Bellman optimality equation, and evaluates it on 12 confounded Atari games.
Problem
Off-policy deep RL from offline demonstrations fails when unobserved confounders violate the no-unmeasured-confounder assumption, making the effect of policies non-identifiable from observed data.
Approach
Extends the Bellman optimality equation with a causal lower bound on Q-values computed from observational transition and reward estimates, using worst-case next-state values when the observed action differs from the evaluated action, and trains a DQN on this bound.
Key results
- Causal-DQN reaches a normalized mean return of 1.04 and normalized IQM of 1.02 relative to the DIAMOND demonstrator across 12 confounded Atari games, versus 0.13 for Interv. DQN and 0.10 for Conf. DQN.
- With the sebulba demonstrator, Causal-DQN reaches a normalized mean of 0.55 and IQM of 0.59.
- Causal-DQN surpasses the demonstrator in 7 of 12 games (DIAMOND setting) per the authors' count.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the standard DQN update with the causal lower-bound update | Nature DQN on confounded Atari games · 12 confounded Atari games | Normalized Mean: 0.13 -> 1.04 | Helped |
| Replace the standard DQN update with the causal lower-bound update | Nature DQN on confounded Atari games · 12 confounded Atari games | Normalized IQM: 0.13 -> 1.02 | Helped |
| Use an LSTM-based DQN instead of CNN-based DQN | DQN trained with confounded demonstrator data · Pong | Average evaluation return: -20.8 -> -20.4 | No clear effect |
| Train directly under masked observations instead of using confounded demonstrations | DQN on masked Pong · Pong | Average evaluation return | No clear effect |
| Use sebulba rather than diamond as the demonstrator for Causal-DQN | Causal-DQN · 12 confounded Atari games | Normalized Mean: 0.53 -> 0.55 | Mixed |
| Train Interv. DQN for longer than 1M environment steps | Interv. DQN on confounded Atari games · Confounded Atari games | Evaluation performance: 3M steps | Helped |
Surprising findings
- Standard DQN variants (interventional, confounded demonstrations, LSTM) largely fail to learn in the masked Atari games within 1M steps, and confounded demonstrations can hurt learning compared to interventional learning.
- Causal-DQN outperforms the demonstrator with full observations in several games despite using masked observations and a smaller CNN.
Limitations
- When demonstrator data has zero support in the unmasked observation area, no meaningful behavior can be learned (e.g., Boxing with sebulba).
- Worst-case next-state is estimated by random sampling of next states, and the method is only instantiated with DQN, not with stronger deep RL algorithms.
- Bounds are conservative; performance depends on the confounded environment design chosen by the authors.
Experimental setup
Datasets: Atari (Gymnasium) confounded games with demonstrator trajectories
Benchmarks: 12 confounded Atari games (Amidar, Asterix, Boxing, Breakout, ChopperCommand, Gopher, KungFuMaster, MsPacman, Pong, Qber
Metrics: average evaluation return normalized mean normalized median normalized IQM
Baselines: Interv. DQN Conf. DQN Conf. LSTM-DQN Random
Builds on: DQN Double DQN Zhang & Bareinboim partial identification of policy values Manski bounds
Closest papers by meaning
- Deep Reinforcement Learning with Double Q-Learning (AAAI 2016 · similarity 0.90)
- Towards Robust Offline Reinforcement Learning under Diverse Data Corruption (ICLR 2024 · similarity 0.89)
- Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding (ICLR 2024 · similarity 0.88)
- Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error (ICML 2024 · similarity 0.87)
- Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach (ICML 2024 · similarity 0.87)
- Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.