Research map / Reinforcement learning / Policy gradient and actor-critic
Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates entropy-regularized max-min multi-objective RL as a two-player zero-sum regularized game and proposes single-loop ERAM/ARAM algorithms (mirror-descent weight updates with PPO learner), with last-iterate convergence in the tabular case and outperformance on traffic signal control.
Problem
Max-min multi-objective reinforcement learning (fairness-oriented) is non-differentiable and prior model-free methods are memory-heavy and only guarantee average-iterate convergence.
Approach
Cast max-min MORL as a two-player zero-sum regularized continuous game between a policy learner and a weight adversary on the simplex; the adversary uses an entropy-regularized mirror-descent update with a closed-form softmax solution, and the learner uses NPG (PPO in deep RL), yielding a single-loop algorithm with last-iterate convergence.
Key results
- ARAM achieves max-min return of -1160 on Base-4 versus -1387 for ERAM and -1838 for GGF-DQN, the best among compared methods.
- ERAM uses about 13,704 parameters for weight updates versus 274,084 for the prior primal-dual method, a roughly 95% reduction.
- ERAM cut training wall time by about 64-68% relative to the prior primal-dual method across three traffic environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ERAM's uniform-reference adversary regularization with ARAM's adaptive correlation-based regularizer (ablation) | ERAM · Base-4 traffic signal control | max-min return: -1387 -> -1160 | Helped |
| Replace ERAM's uniform-reference adversary regularization with ARAM's adaptive correlation-based regularizer (ablation) | ERAM · Asym-4 traffic signal control | max-min return: -2732 -> -2696 | Helped |
| Replace ERAM's uniform-reference adversary regularization with ARAM's adaptive correlation-based regularizer (ablation) | ERAM · Asym-16 traffic signal control | max-min return: -17334 -> -15043 | Helped |
| Vary the adversary update parameter beta away from its endpoint regions (ablation) | ERAM and ARAM · Traffic signal control | max-min return | No clear effect |
| Remove ARAM's adaptive regularizer by setting beta near 1 (ablation) | ARAM · Traffic signal control | max-min return | Mixed |
| Remove ERAM's mirror-descent term by setting beta near 0 (ablation) | ERAM · Traffic signal control | max-min return | Mixed |
| Use ERAM instead of the projected-gradient baseline | ERAM · Base-4 traffic signal control | training wall time (minutes): 346 -> 111 | Helped |
| Use ERAM instead of the projected-gradient baseline | ERAM · Asym-4 traffic signal control | training wall time (minutes): 241 -> 87.2 | Helped |
| Use ERAM instead of the projected-gradient baseline | ERAM · Asym-16 traffic signal control | training wall time (minutes): 1125 -> 356 | Helped |
| Use ARAM instead of the projected-gradient baseline | ARAM · Base-4 traffic signal control | training wall time (minutes): 346 -> 120 | Helped |
| Use ARAM instead of the projected-gradient baseline | ARAM · Asym-4 traffic signal control | training wall time (minutes): 241 -> 87.4 | Helped |
| Use ARAM instead of the projected-gradient baseline | ARAM · Asym-16 traffic signal control | training wall time (minutes): 1125 -> 365 | Helped |
Limitations
- Theoretical last-iterate convergence is proven only for ERAM in the tabular softmax setting; ARAM analysis is left to future work.
- Deep RL experiments use five seeds and few environments.
Experimental setup
Datasets: SUMO-RL traffic signal control simulator MO-Gymnasium environments
Benchmarks: traffic signal control (Base-4, Asym-4, Asym-16)
Metrics: max-min return (minimum of empirical return vector) Nash gap training wall time
Baselines: GGF-DQN GGF-PPO Avg-DQN Gaussian-smoothing primal-dual max-min method
Builds on: PPO GGF-PPO natural policy gradient mirror descent
Closest papers by meaning
- The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm (ICML 2024 · similarity 0.94)
- Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.89)
- Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning (ICML 2024 · similarity 0.88)
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Reward Dimension Reduction for Scalable Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Multi-objective Linear Reinforcement Learning with Lexicographic Rewards (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.