Research map / Reinforcement learning / Visual and representation RL
Preference Controllable Reinforcement Learning with Advanced Multi-Objective Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Preference Controllable RL (PCRL), training a preference-conditioned policy with multi-objective optimization gradient manipulation, plus PreCo, a preference-regularized min-norm update with convergence analysis, evaluated on Fruit-Tree and MO-Reacher.
Problem
Multi-objective RL agents trained by linear scalarization cover only a limited subset of Pareto-optimal solutions and offer poor controllability of the trade-off according to a user preference.
Approach
Train a single preference-conditioned policy by plugging advanced MOO gradient manipulation into RL, and add a similarity gradient (based on a max-ratio-scaled distance between the preference and value vector) to each objective gradient in a min-norm update (PreCo) so the agent both reaches the Pareto front and aligns with the preference.
Key results
- On Fruit-Tree 6-D, PreCo achieves HV 15.61 ± 0.75 and CS 0.78 ± 0.03, versus EPO 14.97 ± 2.29 / 0.77 and LS 5.74 ± 0.88 / 0.72.
- On Fruit-Tree 6-D, PDMORL achieves HV 9.30 and CS 0.89 but is weaker on HV, while CAPQL reaches HV 5.95 and CS 0.72.
- LS and SDMgrad learn only a single equidistant local optimum on MO-Reacher, whereas PreCo and EPO produce preference-specific state coverage.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PCRL with PreCo instead of EPO | PCRL · Fruit-Tree 4D | HV: 1.04 ± 0.05 -> 1.09 ± 0.02 | Helped |
| Use PCRL with PreCo instead of EPO | PCRL · Fruit-Tree 4D | CS: 0.89 ± 0.02 -> 0.91 ± 0.01 | Helped |
| Use PCRL with PreCo instead of EPO | PCRL · Fruit-Tree 5D | HV: 3.98 ± 0.48 -> 4.33 ± 0.21 | Helped |
| Use PCRL with PreCo instead of EPO | PCRL · Fruit-Tree 5D | CS: 0.86 ± 0.03 -> 0.87 ± 0.01 | Helped |
| Use PCRL with PreCo instead of EPO | PCRL · Fruit-Tree 6D | HV: 14.97 ± 2.29 -> 15.61 ± 0.75 | Helped |
| Use PCRL with PreCo instead of EPO | PCRL · Fruit-Tree 6D | CS: 0.77 ± 0.03 -> 0.78 ± 0.03 | Helped |
| Replace LS with PCRL using PreCo | Fruit-Tree · Fruit-Tree | HV: 5.74 ± 0.88 -> 15.61 ± 0.75 | Helped |
| Replace LS with PCRL using PreCo | Fruit-Tree · Fruit-Tree | CS: 0.72 ± 0.01 -> 0.78 ± 0.03 | Helped |
| Replace PDMORL with PCRL using PreCo | Fruit-Tree · Fruit-Tree | HV: 9.30 ± 0.08 -> 15.61 ± 0.75 | Helped |
| Replace PDMORL with PCRL using PreCo | Fruit-Tree · Fruit-Tree | CS: 0.89 ± 0.05 -> 0.78 ± 0.03 | Hurt |
| Replace CAPQL with PCRL using PreCo | Fruit-Tree · Fruit-Tree | HV: 5.95 ± 1.12 -> 15.61 ± 0.75 | Helped |
| Replace CAPQL with PCRL using PreCo | Fruit-Tree · Fruit-Tree | CS: 0.72 ± 0.02 -> 0.78 ± 0.03 | Helped |
Surprising findings
- PDMORL attains higher cosine similarity than PreCo on Fruit-Tree (0.89 vs 0.78) yet much lower hypervolume.
- CAGrad fails to learn meaningful results on MO-Reacher due to stochastic gradient variance.
Limitations
- Continuous benchmarks MO-Hopper and MO-Ant have symmetric objectives where low-level RL implementation matters more than algorithm design.
- Converged stationary points do not guarantee high similarity; some preferences may correspond to unreachable Pareto regions.
- Theory relies on increasing or constant lambda with an upper limit in practice.
Experimental setup
Datasets: Fruit-Tree MO-Reacher MO-Hopper MO-Ant
Benchmarks: Fruit-Tree MO-Reacher MO-Hopper MO-Ant
Metrics: hypervolume cosine similarity
Baselines: Linear Scalarization EPO CAGrad SDMGrad CAPQL PDMORL GPI-LS
Builds on: MGDA EPO CAGrad SDMGrad PMTL CAPQL
Closest papers by meaning
- Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.91)
- Pareto Set Learning for Multi-Objective Reinforcement Learning (AAAI 2025 · similarity 0.89)
- Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF (ICML 2025 · similarity 0.88)
- Preference Optimization on Pareto Sets: On a Theory of Multi-Objective Optimization (NeurIPS 2025 · similarity 0.88)
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.88)
- An Analytical Study of Utility Functions in Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.