Research map / Reinforcement learning / Policy gradient and actor-critic
Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution
Zhenshuo Zhang, Ruoxuan Xiong, Hongyang Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multi-objective RL approach using routed ensembles and trajectory attribution to learn policies that balance competing objectives across preference weights.
Problem
Multi-objective reinforcement learning must learn policies that balance competing objectives, and optimal policies can vary sharply with the preferred objective weights.
Approach
Uses routed ensembles with trajectory attribution to handle sharp variation of optimal policies across objective weights; the abstract is truncated so the specific mechanism is only partly stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| route trajectories to basis policies by highest scalarized return | multi-task RL base learner · Meta-World MT10 | success rate (%): 82.0 -> 85.6 | Improved |
| route trajectories to the best basis policy rather than assigning uniformly at random (ablation) | routed ensemble · Meta-World MT10 | success rate (%): 81.3 -> 85.6 | Improved |
| route trajectories instead of uniform assignment (ablation) | routed ensemble · MO-HalfCheetah2d | HV (x10^6): 3.17 -> 5.17 | Improved |
| route trajectories instead of uniform assignment (ablation) | routed ensemble · MO-Hopper | HV (x10^7): 1.56 -> 1.83 | Improved |
| route trajectories instead of uniform assignment (ablation) | routed ensemble · Fruit-Tree | HV (x10^3): 8.29 -> 9.30 | Improved |
| route trajectories instead of uniform assignment (ablation) | routed ensemble · RLHF, Qwen3-1.7B HelpSteer2 | hypervolume: 0.47 -> 0.54 | Improved |
| route trajectories instead of uniform assignment (ablation) | routed ensemble · RLHF, Llama-3.2-1B HelpSteer2 | hypervolume: 0.35 -> 0.35 | No clear effect |
| use a routed ensemble instead of a single policy | routed ensemble · synthetic RLHF environment | hypervolume: 0.48 -> 0.56 | Improved |
| add routing across an ensemble of basis policies | policy ensemble · synthetic environment | coverability (chi^2 divergence): 1.3 -> 0.2 | Improved |
| increase number of basis policies N from 2 to 5 (ablation) | routed ensemble · MO-Gymnasium MuJoCo | HV: best N=3 for HalfCheetah (5.21) and Hopper (1.87), N=5 for Ant (2.12) | Mixed |
| use nonlinear (quadratic, cubic, RBF) kernel instead of linear kernel (ablation) | kernel surrogate attribution · synthetic dataset | fitting loss: 0.2 -> 0.1 | Improved |
| use kernel surrogate instead of linear surrogates for attribution | trajectory attribution · Synergy MTRL | LDS: 0.16 -> 0.91 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- Q-Pensieve — Q-Pensieve: Boosting Sample Efficiency of Multi-Objective RL Through Memory Sharing of Q-Snapshots (ICLR 2023) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.88)
- Multi-objective Linear Reinforcement Learning with Lexicographic Rewards (ICML 2025 · similarity 0.88)
- Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning (ICML 2024 · similarity 0.87)
- Pareto Set Learning for Multi-Objective Reinforcement Learning (AAAI 2025 · similarity 0.87)
- Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
