Research map / Reinforcement learning / Policy gradient and actor-critic
Conflict-Averse Gradient Aggregation for Constrained Multi-Objective Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoMOGA, a constrained multi-objective RL method that turns objective maximization into a constrained QP over linearized gradients so that updates avoid gradient conflicts, handling safety constraints without Lagrange multipliers.
Problem
Constrained multi-objective RL where linear scalarization with deep function approximation converges to local optima due to gradient conflicts, and Lagrangian methods for safety constraints are unstable.
Approach
Treat the multi-objective maximization as a constrained optimization problem where each weighted objective must not decrease (linearized), plus existing safety constraints; solve the resulting QP for an aggregated gradient, avoiding gradient conflicts and extra Lagrange variables, and distill per-preference policies into a universal preference-conditioned policy.
Key results
- CoMOGA achieves the highest hypervolume across all legged locomotion tasks while satisfying all constraints, with the lowest sparsity in those tasks.
- On Safety Gymnasium, CoMOGA shows the highest HV in all four tasks and the lowest SP in two of four, while satisfying all constraints.
- On MO-Gymnasium, CoMOGA achieves the highest HV in three of six tasks and the lowest SP in five of six tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CoMOGA instead of the baselines | Legged robot locomotion · Bipedal and quadrupedal tasks | hypervolume (HV) | Helped |
| use CoMOGA instead of the baselines | Legged robot locomotion · Bipedal and quadrupedal tasks | sparsity (SP) | Helped |
| use CoMOGA instead of the baselines | Safety Gymnasium · Single-agent and multi-agent goal tasks | hypervolume (HV) | Helped |
| use CoMOGA instead of the baselines | Safety Gymnasium · Single-agent and multi-agent goal tasks | sparsity (SP) | Helped |
| use CoMOGA instead of the baselines | Multi-Objective Gymnasium · Six MuJoCo tasks | hypervolume (HV) | Helped |
| use CoMOGA instead of the baselines | Multi-Objective Gymnasium · Six MuJoCo tasks | sparsity (SP) | Helped |
| combine CAGrad with the Lagrangian approach instead of using CoMOGA (ablation) | Legged robot locomotion · Quadrupedal task | constraint satisfaction | Hurt |
| combine CAGrad with the Lagrangian approach instead of using CoMOGA (ablation) | Legged robot locomotion · Bipedal task | hypervolume (HV) | Hurt |
| combine CAGrad with the Lagrangian approach instead of using CoMOGA (ablation) | Legged robot locomotion · Bipedal task | sparsity (SP) | Hurt |
Surprising findings
- Walker2d appears to be solved by all algorithms except LP3 with similarly high HV, suggesting reward design may let baselines reach near-global optima.
Limitations
- Policies may cover only a narrow portion of the CP front because updates forbid any objective from decreasing.
- Convergence guarantee holds in tabular settings with a Fisher-matrix variant; deep-network practical version is not theoretically guaranteed.
Experimental setup
Datasets: Safety Gymnasium legged robot locomotion tasks MO-Gymnasium MuJoCo tasks
Benchmarks: Safety Gymnasium legged robot locomotion MO-Gymnasium MuJoCo
Metrics: hypervolume sparsity normalized sparsity constraint satisfaction
Baselines: LP3 PD-MORL CAPQL CAGrad+Lagrangian linear scalarization
Builds on: TRPO SAC CAGrad PD-MORL
Closest papers by meaning
- The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm (ICML 2024 · similarity 0.87)
- Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.87)
- Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF (ICML 2025 · similarity 0.87)
- Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning (ICML 2024 · similarity 0.87)
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.87)
- An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.