Research map / Reinforcement learning / MDPs and planning
Policy Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes multi-agent value alignment as policy aggregation and adapts social choice rules (proportional veto core, quantile fairness, alpha-approval, Borda count) to the continuous state-action occupancy polytope, with hardness results and MILP/LP implementations.
Problem
Aggregating multiple agents' different reward functions (and hence optimal policies) in a shared MDP into a desirable collective policy, while being invariant to affine transformations of rewards.
Approach
Reinterpret ordinal preferences over policies as volumes of subsets of the state-action occupancy polytope, so that social choice rules (veto core, quantile fairness, alpha-approval, Borda) apply to policy aggregation and are invariant to positive affine reward transformations.
Key results
- Max-quantile achieves the lowest Gini index (0.0188 vs 0.2864 for egalitarian and 0.4392 for utilitarian) in the 5 symmetric agents scenario.
- Borda and max-quantile give Nash welfare of about 0.536 in the 5-agent scenario, versus 0.050 for utilitarian.
- Proportional veto core is nonempty and computable in polynomial time; a (1/e)-quantile fair policy always exists and the bound is tight.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use max-quantile instead of utilitarian aggregation | Policy aggregation · 5 symmetric agents, one per warehouse | Gini index: 0.4392 ± 0.0094 -> 0.0188 ± 0.0022 | Helped |
| use max-quantile instead of utilitarian aggregation | Policy aggregation · 10 agents, random subsets of warehouses | Gini index: 0.2020 ± 0.0182 -> 0.0625 ± 0.0067 | Helped |
| use max-quantile instead of Borda aggregation | Policy aggregation · 5 symmetric agents, one per warehouse | Gini index: 0.0225 ± 0.0024 -> 0.0188 ± 0.0022 | Helped |
| use Borda instead of max-quantile aggregation | Policy aggregation · 5 symmetric agents, one per warehouse | Nash welfare: 0.5355 ± 0.0062 -> 0.5356 ± 0.0062 | No clear effect |
| use Borda instead of max-quantile aggregation | Policy aggregation · 10 agents, random subsets of warehouses | Nash welfare: 0.6474 ± 0.0232 -> 0.6801 ± 0.0261 | Helped |
Surprising findings
- Plurality ignores almost all agents and is hard to approximate (no polynomial 1/n^{1/2-eps} approximation unless P=NP).
- Utilitarian and egalitarian rules, being sensitive to reward scales, perform unfairly.
Limitations
- Computation may not scale beyond thousands of agents, states, and actions.
- Rules are likely manipulable (Gibbard-Satterthwaite); strategyproofness conjectured to fail.
- Assumes known MDP and reward functions; online/continuous settings left as future work.
Experimental setup
Datasets: simulated dynamic attention allocation MOMDP (warehouse monitoring, 3^5 states)
Benchmarks: warehouse monitoring MOMDP (5 symmetric agents; 10 random-subset agents)
Metrics: normalized expected return per agent Gini index Nash welfare utilitarian welfare
Baselines: utilitarian welfare egalitarian welfare alpha-approval (80%, 90%) Borda count max-quantile
Builds on: proportional veto core quantile fairness (Procaccia et al. discrete fair division) Borda count alpha-approval voting dynamic attention allocation environment
Closest papers by meaning
- AI Alignment with Changing and Influenceable Reward Functions (ICML 2024 · similarity 0.89)
- Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.89)
- Can an AI Agent Safely Run a Government? Existence of Probably Approximately Aligned Policies (NeurIPS 2024 · similarity 0.87)
- Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learning (ICML 2025 · similarity 0.87)
- Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking (ICLR 2025 · similarity 0.87)
- Fair Participation via Sequential Policies (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.