Research map / Reinforcement learning / Policy gradient and actor-critic

PA2D-MORL: Pareto Ascent Directional Decomposition Based Multi-Objective Reinforcement Learning

AAAI 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PA2D-MORL, a multi-policy MORL method using Pareto ascent directions for weight-free policy gradient decomposition, partitioned greedy randomized selection, and Pareto adaptive fine-tuning, evaluated on seven MuJoCo multi-objective tasks.

Problem

Approximating the Pareto policy set in multi-objective reinforcement learning for complex continuous or high-dimensional control tasks, where prior multi-policy methods rely on prediction models or hand-chosen preferences and yield unevenly distributed or unstable policy sets.

Approach

Uses the Pareto ascent direction (minimum-norm common ascent direction from multi-gradient descent) to set scalarization weights for each policy's PPO update, avoiding prediction models and prior preferences; combines this with partitioned greedy randomized policy selection in an evolutionary population and a Pareto adaptive fine-tuning step that targets large gaps in the front.

Key results

• PA2D-MORL achieves the highest HV in all seven MuJoCo multi-objective environments, e.g., Walker2d HV 5.743e6 vs PGMORL 4.849e6 and Humanoid 51.23e6 vs 44.75e6.
• PA2D-MORL gives the lowest SP (denser front) in most environments, except HalfCheetah where PGMORL is slightly better (SP 0.022e-4 vs 0.026e-4).
• Ablating PA-FT (PA2D-MORL-ablated) lowers HV and raises SP, e.g., Walker2d SP 0.180e-4 vs 0.014e-4, showing PA-FT improves front density.

Measured effects reported in the paper

Surprising findings

• Increasing the number of fine-tuning generations to 4/5 of training significantly degrades performance, so PA-FT must be introduced early.

Limitations

• Theoretical analysis of Pareto-stationarity and ascent direction is based on convex optimization; non-convex settings may converge to local optima.
• Weighted-sum decomposition cannot reach non-convex Pareto front parts directly; relies on PA-FT and dominance-based set maintenance to cover them.
• The modified PFA baseline could not be applied to the 3-objective Hopper-3 task, so comparison there is incomplete.

Experimental setup

Datasets: MuJoCo multi-objective variants (Walker2d, Humanoid, HalfCheetah, Hopper-2, Ant, Swimmer, Hopper-3)

Benchmarks: Walker2d Humanoid HalfCheetah Hopper-2 Ant Swimmer Hopper-3

Metrics: hypervolume (HV) sparsity (SP)

Baselines: PGMORL MOEA/D PFA PA2D-MORL-ablated

Builds on: PPO PGMORL Pareto policy adaptation MOEA/D multi-gradient descent

Closest papers by meaning

• Pareto Set Learning for Multi-Objective Reinforcement Learning (AAAI 2025 · similarity 0.91)
• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.88)
• Preference Controllable Reinforcement Learning with Advanced Multi-Objective Optimization (ICML 2025 · similarity 0.87)
• On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.87)
• Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning (ICML 2024 · similarity 0.87)
• An Analytical Study of Utility Functions in Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.86)