Research map / Reinforcement learning / MDPs and planning
Personalized Reinforcement Learning with a Budget of Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes represented MDPs (r-MDPs) where a small budget of k policies serves n users, and two K-means/EM-inspired and end-to-end deep RL (PPO-based) algorithms that jointly learn user-to-policy assignments and policies to maximize social welfare.
Problem
Personalizing sequential decision policies for many users when regulatory approval limits the number of deployable policies to a small budget k < n, while maximizing utilitarian social welfare.
Approach
Formalize personalization under a policy budget as an r-MDP in which agents are assigned to k representatives each running one policy; solve it via a factorized approach with an EM-like hard-assignment algorithm (Q-table greedy reassignment plus PPO M-steps) and a soft-assignment end-to-end algorithm that learns assignment logits by backpropagating the PPO loss.
Key results
- On Resource Gathering with n=25, the algorithms reach social welfare above 90 with k=10 and above 85 with k=3, versus an optimum of 93.6 at k=25.
- The clustering baseline shows minimal personalization, with performance largely flat across k in Resource Gathering.
- On MuJoCo tasks the EM and end-to-end algorithms consistently outperform the trajectory-clustering baseline across policy budgets, except on Ant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EM-like or end-to-end assignment learning instead of trajectory clustering | r-MDP policies · Resource Gathering, intermediate k | social welfare | Helped |
| Use EM-like or end-to-end assignment learning instead of trajectory clustering | r-MDP policies · MuJoCo tasks, various policy budgets | social welfare | Helped |
| Use EM-like or end-to-end assignment learning instead of trajectory clustering | r-MDP policies · Ant | social welfare | Mixed |
| Increase policy budget from one policy to an intermediate budget | EM and end-to-end · Resource Gathering | social welfare: above 85; above 90; 93.6 | Helped |
| Use the same single-policy or one-policy-per-agent setting across algorithms | all algorithms · Resource Gathering and MuJoCo | social welfare | No clear effect |
Surprising findings
- The clustering baseline underperforms even though it is a personalized method, because its unanimous policy yields identical rewards for all agents, giving no signal for clustering.
- The clustering baseline beats the proposed algorithms on Ant, the only environment where it outperforms them.
Limitations
- Utilitarian social welfare may produce unfair reward distributions across agents.
- Only the Manufacturing stage is studied; Assessment and Deployment stages are not addressed.
- Experiments are simulated only, with no real-world validation.
Experimental setup
Datasets: Resource Gathering (grid world, adapted) MuJoCo HalfCheetah MuJoCo Ant MuJoCo Hopper MuJoCo Walker2d
Benchmarks: Resource Gathering MuJoCo HalfCheetah MuJoCo Ant MuJoCo Hopper MuJoCo Walker2d
Metrics: social welfare standard error
Baselines: clustering-based personalized RL (trajectory K-means baseline) random assignment
Builds on: PPO K-means Expectation-Maximization generalized advantage estimation
Closest papers by meaning
- Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks (ICML 2025 · similarity 0.89)
- Learning Personalized Decision Support Policies (AAAI 2025 · similarity 0.86)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.85)
- Bridging State and History Representations: Understanding Self-Predictive RL (ICLR 2024 · similarity 0.85)
- On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations (ICLR 2025 · similarity 0.85)
- Oracle-Efficient Reinforcement Learning for Max Value Ensembles (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.