Research map / Reinforcement learning / Policy gradient and actor-critic

Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes D³PO, a PPO-based single preference-conditioned MORL policy that clips per-objective advantages before preference weighting (Late-Stage Weighting) and adds a KL-distance diversity regularizer, reporting higher hypervolume and expected utility on MO-MuJoCo and Building-9d.

Problem

Single preference-conditioned MORL policies are brittle and often fail to recover dense Pareto fronts due to destructive advantage cancellation from early scalarization and representational mode collapse across preferences.

Approach

Apply PPO clipping independently to each per-objective advantage from a multi-head critic and only then combine them with preference weights (Late-Stage Weighting), plus a diversity loss tying the KL divergence between action distributions for nearby preferences to their L1 distance.

Key results

• On Humanoid-2d, D3PO reaches HV 4.03e5 and EU 556 vs. best baseline C-MORL HV 2.08e5 and EU 354 (table values).
• On Ant-3d, D3PO HV 3.65e7 vs. CAPQL 1.08e7 and C-MORL 0.99e7.
• Ablation on Humanoid-2d: removing LSW drops HV from 3.76e5 to 1.50e5; removing diversity loss drops HV to 2.32e5 with SP 0 (collapse).

Surprising findings

• Authors report that the diversity regularizer with alpha=0 drives collapse, and that baselines' sparsity of 0 on Humanoid-2d indicates total collapse to one policy.
• D3PO has higher sparsity than baselines on Ant-2d because its hypervolume is much larger, which the authors attribute to spread-out solutions.

Limitations

• Operates only within linear scalarization, so cannot reach non-convex (concave) Pareto front regions, as the authors acknowledge.
• Sparsity metric is noisy and D3PO's higher SP on Ant-2d is attributed to larger hypervolume rather than poorer coverage.
• Baseline numbers are re-run with v5 environments and differ from original papers; ablations use a reduced 1.5e6-step budget.

Experimental setup

Datasets: MO-Gymnasium MO-MuJoCo (Hopper, Ant, Humanoid) Building-9d Minecart Lunar Lander-4d Fruit Tree

Benchmarks: MO-Hopper-2d MO-Hopper-3d MO-Ant-2d MO-Ant-3d MO-Humanoid-2d Building-9d Minecart Lunar Lander-4d

Metrics: hypervolume expected utility sparsity

Baselines: CAPQL MORL/D PG-MORL PreCo C-MORL GPI-LS

Builds on: PPO MORL/D CAPQL C-MORL PG-MORL

Closest papers by meaning

• On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.91)
• Preference Controllable Reinforcement Learning with Advanced Multi-Objective Optimization (ICML 2025 · similarity 0.91)
• MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning (ICML 2025 · similarity 0.91)
• FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Pareto Set Learning for Multi-Objective Reinforcement Learning (AAAI 2025 · similarity 0.90)
• MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment (ICML 2025 · similarity 0.89)