Research map / Reinforcement learning / Policy gradient and actor-critic
Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes alpha-approximate policy portfolios covering all generalized p-means (p ≤ 1) in multi-objective RL, with a p-MeanPortfolio algorithm, a budget-constrained heuristic, and a warm-start analysis, evaluated on taxi, disaster resource allocation, and healthcare RMAB settings.
Problem
Choosing a social welfare function (generalized p-mean) for aggregating multiple stakeholder rewards in RL is hard because optimal policies can be highly sensitive to p, so a decision-maker needs a small set of policies covering all p ≤ 1.
Approach
Construct a finite alpha-approximate portfolio of policies by sequentially selecting p values via a line search so that each selected policy is alpha-approximately optimal over an interval of p, with bounds on portfolio size and oracle calls; plus a budget-constrained heuristic that greedily refines p-intervals.
Key results
- p-MeanPortfolio achieves approximation 0.999 with portfolio size 3 on the natural disaster task and 0.982 with size 2 on healthcare intervention.
- Random p sampling and random policy sampling baselines reach at most about 0.96 and 0.71 respectively at matched sizes.
- BudgetConstrainedPortfolio matches p-MeanPortfolio quality closely (e.g., 0.921 at size 2-4 on disaster) using far fewer oracle calls (e.g., 4 vs 50 at size 4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Construct a portfolio with p-MeanPortfolio instead of random policy or random p sampling | MORL portfolios · Resource Allocation after Natural Disaster | actual approximation: 0.706 vs 0.832 at portfolio size 1; 0.904 vs 0.890 at size 2; 0.999 vs 0.892 at size 3; 1.000 vs 0.888 at size 4 | Mixed |
| Construct a portfolio with p-MeanPortfolio instead of random policy or random p sampling | MORL portfolios · Healthcare Intervention | actual approximation: 0.924 vs 0.913; 0.982 vs 0.941; 0.982 vs 0.929; 0.982 vs 0.947; 0.993 vs 0.957; 0.999 vs 0.962; 1.000 vs 0.953 | Helped |
| Construct a portfolio with p-MeanPortfolio instead of random policy or random p sampling | MORL portfolios · Taxi Environment | actual approximation: 0.66 vs 0.66; 0.61 vs 0.61; 0.97 vs 0.54; 0.87 vs 0.71; 0.99 vs 0.60 | Mixed |
| Use Budget-ConstrainedPortfolio instead of p-MeanPortfolio at the same portfolio size | MORL portfolios · Resource Allocation after Natural Disaster | actual approximation: 0.706 vs 0.885; 0.904 vs 0.921; 0.999 vs 0.921; 1.000 vs 0.921 | Mixed |
| Use Budget-ConstrainedPortfolio instead of p-MeanPortfolio at the same portfolio size | MORL portfolios · Healthcare Intervention | actual approximation: 0.924 vs 0.938; 0.982 vs 0.938; 0.982 vs 0.938; 0.982 vs 0.938; 0.993 vs 0.986; 0.999 vs 0.986; 1.000 vs 0.993 | Mixed |
| Use Budget-ConstrainedPortfolio instead of p-MeanPortfolio at the same portfolio size | MORL portfolios · Taxi Environment | actual approximation: 0.66 vs 0.66; 0.61 vs 0.61; 0.97 vs 0.92; 0.87 vs 0.90; 0.99 vs 0.92 | Mixed |
| Use Budget-ConstrainedPortfolio instead of p-MeanPortfolio | MORL portfolio construction · Resource Allocation after Natural Disaster | oracle calls: 7 vs 2; 18 vs 3; 50 vs 4 | Helped |
| Use Budget-ConstrainedPortfolio instead of p-MeanPortfolio | MORL portfolio construction · Healthcare Intervention | oracle calls: 2 vs 1; 7 vs 2; 11 vs 3; 19 vs 4; 23 vs 5; 46 vs 6; 61 vs 7 | Helped |
| Use Budget-ConstrainedPortfolio instead of p-MeanPortfolio | MORL portfolio construction · Taxi Environment | oracle calls: 17 vs 1; 30 vs 2; 44 vs 3; 118 vs 8; 144 vs 10 | Helped |
Surprising findings
- The budget-constrained heuristic sometimes outperforms p-MeanPortfolio at very small portfolio sizes (K=1 or 2) because it selects boundary p values first.
- Random p sampling occasionally matches or beats p-MeanPortfolio in the taxi environment.
Limitations
- Achieved approximation is measured against what the RL algorithm can compute, not true optima, and RL randomness can cause deviations (e.g., size-8 taxi portfolio at 0.87 despite alpha=0.9).
- Theoretical guarantees assume an oracle for the optimal policy, which is hard to achieve exactly in RL.
- Budget-constrained heuristic relies on approximations (interval monotonicity assumption) without formal guarantees.
Experimental setup
Datasets: taxi grid world (synthetic) natural disaster resource allocation (synthetic) ARMMAN maternal mHealth program data (healthcare RMAB)
Benchmarks: taxi environment natural disaster resource allocation healthcare intervention RMAB
Metrics: actual approximation ratio portfolio size number of oracle calls
Baselines: random p sampling random policy sampling GPI BudgetConstrainedPortfolio
Builds on: Welfare Q-Learning Generalized Policy Improvement
Closest papers by meaning
- Policy Aggregation (NeurIPS 2024 · similarity 0.87)
- Preference Optimization on Pareto Sets: On a Theory of Multi-Objective Optimization (NeurIPS 2025 · similarity 0.86)
- Explicit Exploration for High-Welfare Equilibria in Game-Theoretic Multiagent Reinforcement Learning (ICML 2025 · similarity 0.86)
- An Analytical Study of Utility Functions in Multi-Objective Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.