Research map / Reinforcement learning / Visual and representation RL
Pareto Set Learning for Multi-Objective Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PSL-MORL, a decomposition-based multi-objective RL framework where a hypernetwork maps preference vectors to policy network parameters (fused with a main policy network), compatible with DDQN and TD3, with theory on Rademacher complexity and Pareto set learning.
Problem
Obtaining a dense, personalized approximation of the entire Pareto front in multi-objective RL, where prior methods either fail to cover the front or use a single policy network for all preferences.
Approach
Use a hypernetwork as the Pareto set model: given a preference weight it generates policy parameters, which are blended with a main policy network via parameter fusion for stable training; the framework wraps any single-objective RL algorithm.
Key results
- On MO-MuJoCo, PSL-MORL achieves the best average rank on hypervolume (1.8) and sparsity (1.6) across five environments, with PD-MORL second.
- On MO-Ant-v2, PSL-MORL reaches hypervolume 8.63 (x10^6) versus 7.48 for PD-MORL.
- On Fruit Tree Navigation d=7, PSL-MORL hypervolume is 11786.10 versus 11419.58 for PD-MORL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PSL-MORL instead of PD-MORL | TD3-based MORL · MO-Ant-v2 | HV (×10^6): 7.48 -> 8.63 | Helped |
| Use PSL-MORL instead of PD-MORL | TD3-based MORL · MO-Walker2d-v2 | HV (×10^6): 5.41 -> 5.36 | Hurt |
| Use PSL-MORL instead of the strongest baseline | TD3-based MORL · MO-HalfCheetah-v2 | SP (×10^3): 0.44 -> 0.16 | Helped |
| Use PSL-MORL instead of the strongest baseline | TD3-based MORL · MO-Ant-v2 | SP (×10^4): 0.37 -> 0.68 | Hurt |
| Use PSL-MORL instead of PD-MORL | DDQN-based MORL · FTN d=6 | Hypervolume: 9299.15 -> 9302.38 | Helped |
| Use PSL-MORL instead of PD-MORL | DDQN-based MORL · FTN d=5 | Sparsity: N/A -> 0.01 | Helped |
| Remove parameter fusion and use only hypernetwork-generated parameters (ablation) | PSL-MORL · MO-MuJoCo | HV: 5.36 vs 1.47 (MO-Walker2d-v2); 5.92 vs 4.65 (MO-HalfCheetah-v2); 8.63 vs 1.39 (MO-Ant-v2); 3.22 vs 2.11 (MO-Swimmer-v2); | Hurt |
| Directly add hypernetwork-generated parameters to primary policy parameters (ablation) | PSL-MORL · MO-MuJoCo | HV: 5.36 vs 2.63 (MO-Walker2d-v2); 5.92 vs 5.62 (MO-HalfCheetah-v2); 8.63 vs 4.27 (MO-Ant-v2); 3.22 vs 2.67 (MO-Swimmer-v2); | Hurt |
| Vary parameter fusion coefficient α (ablation) | PSL-MORL · MO-MuJoCo | HV: MO-Walker2d-v2: 5.36 at 0.01; MO-HalfCheetah-v2: 5.92 at 0.01; MO-Ant-v2: 8.63 at 0.01; MO-Swimmer-v2: 3.22 at 0.01; MO- | Mixed |
Surprising findings
- Using only the hypernetwork-generated parameters (PSL-MORL-gen) or directly adding them to the main network (PSL-MORL-add) drops performance drastically, showing parameter fusion is critical.
- PSL-MORL does not win every column; PD-MORL has a better hypervolume on MO-Walker2d-v2 and MOEA/D on MO-Hopper-v2.
Limitations
- Only linear scalarization is considered; non-linear utility functions are left for future work.
- Theoretical superiority is shown via Rademacher complexity bounds, which do not directly guarantee empirical performance.
Experimental setup
Datasets: MO-MuJoCo Fruit Tree Navigation
Benchmarks: MO-MuJoCo Fruit Tree Navigation
Metrics: hypervolume sparsity
Baselines: PG-MORL PD-MORL Radial Algorithm Pareto Front Adaption MOEA/D RANDOM META Envelope Q-learning
Builds on: Pareto Set Learning DDQN TD3 PD-MORL Hindsight Experience Replay
Closest papers by meaning
- PA2D-MORL: Pareto Ascent Directional Decomposition Based Multi-Objective Reinforcement Learning (AAAI 2024 · similarity 0.91)
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.90)
- Preference Optimization on Pareto Sets: On a Theory of Multi-Objective Optimization (NeurIPS 2025 · similarity 0.89)
- Preference Controllable Reinforcement Learning with Advanced Multi-Objective Optimization (ICML 2025 · similarity 0.89)
- An Analytical Study of Utility Functions in Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.