Research map / Multi-agent systems and games / Games and equilibria
Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames robust RL as zero-sum positional differential games and proposes Isaacs DQN (IDQN) and decomposed IDQN (DIDQN), a centralized shared-Q-function method justified by Isaacs's condition, which outperforms RRL and MARL baselines on eight zero-sum differential games.
Problem
Robust RL with an adversarial second agent is a zero-sum game; Markov game theory gives only mixed-policy values, so deterministic guaranteed payoffs and a shared Q-function for centralized learning lack theoretical grounding.
Approach
Under Isaacs's condition, a single shared Q-function approximately solves both minimax and maximin Bellman equations in discretized positional differential games, enabling pure-policy centralized Q-learning (IDQN), and it can be decomposed into agent-specific terms when dynamics and running cost separate (DIDQN).
Key results
- IDQN and DIDQN show the best guaranteed results across all eight differential games, with DIDQN clearly better on InvertedPendulum.
- 2xDDQN, RARL and NashDQN perform worst; the paper attributes this to overfitting to a specific opponent and to stochastic mixed policies.
- MADDPG achieves satisfactory guaranteed results only on GetIntoCircle and GetIntoSquare.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use IDQN or DIDQN instead of 2xDDQN, RARL, or NashDQN | zero-sum differential games · Eight differential games | guaranteed-result performance | Helped |
| use MADQN instead of 2xDDQN, RARL, or NashDQN | zero-sum differential games · Eight differential games | performance | Helped |
| use IDQN or DIDQN instead of MADQN | zero-sum differential games · Eight differential games | performance | Helped |
| use CounterDQN instead of MADQN | zero-sum differential games · Almost all games | performance | Hurt |
| use CounterDQN instead of MADQN | zero-sum differential games · HomicidalChauffeur and InvertedPendulum | performance | Mixed |
| decompose the shared Q-function | DIDQN · InvertedPendulum | performance | Helped |
| use IDQN or DIDQN rather than decentralized learning | zero-sum differential games · Eight differential games | robustness and performance | Helped |
Surprising findings
- CounterDQN, where the second agent knows the first agent's next action, performs worse than MADQN in almost all games.
- Isaacs's condition does not guarantee a Nash equilibrium in discrete-time games, as shown by a counterexample.
Limitations
- Theory requires Isaacs's condition, which may fail in some games; the paper notes MADQN is then more theoretically justified.
- Algorithms operate on finite discretized action spaces, which may be limiting for high-dimensional action spaces.
- No general existence of equilibrium in feedback policies for multi-agent differential games beyond Isaacs's condition.
Experimental setup
Datasets: EscapeFromZero GetIntoCircle GetIntoSquare HomicidalChauffeur Interception InvertedPendulum Swimmer HalfCheetah
Benchmarks: EscapeFromZero GetIntoCircle GetIntoSquare HomicidalChauffeur Interception InvertedPendulum Swimmer HalfCheetah
Metrics: guaranteed result (quality index J) exploitability
Baselines: 2xDDQN RARL MADDPG NashDQN MADQN CounterDQN
Builds on: DQN RARL NashDQN MADDPG
Closest papers by meaning
- Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations (ICLR 2024 · similarity 0.87)
- Confounding Robust Deep Reinforcement Learning: A Causal Approach (NeurIPS 2025 · similarity 0.87)
- Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error (ICML 2024 · similarity 0.87)
- Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula (ICLR 2024 · similarity 0.87)
- Improve Robustness of Reinforcement Learning against Observation Perturbations via l∞ Lipschitz Policy Networks (AAAI 2024 · similarity 0.87)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.