Research map / Reinforcement learning / Policy gradient and actor-critic
An Analytical Study of Utility Functions in Multi-Objective Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of utility functions in multi-objective RL, giving conditions under which utility-optimal stationary policies exist and characterising which preference relations can be represented by (quasi-representative) utility functions.
Problem
MORL lacks theoretical characterisation of which utility functions admit an optimal policy and which user preference relations can be expressed as utility functions, although its solution concepts implicitly assume both.
Approach
Introduces formal definitions of per-state and all-states utility optimality, preference relations between policies, and quasi-representative utility functions; proves continuity suffices for per-state optimal stationary policies, affine-composed-with-strictly-increasing suffices for global deterministic optimal policies, and complete transitive preferences with maximal elements admit quasi-repres
Key results
- Theorem 1: for continuous utility functions and any finite MOMDP, at least one stationary per-state utility-optimal policy exists for every state.
- Theorem 2: utilities of the form h(g(x)) with g affine and h strictly monotonically increasing admit at least one deterministic stationary u-optimal policy in finite MOMDPs.
- Theorem 3: complete, transitive preferences with a maximal element at every state admit a quasi-representative utility function in any finite MOMDP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| require the utility function to be continuous over all policy value functions | finite MOMDP · each state | state-level utility-optimal policy existence | Helped |
| require the utility to be a strictly increasing function of an affine scalarisation | finite MOMDP · all states | global utility-optimal policy existence | Helped |
| use a monotonically increasing Chebyshev utility | finite MOMDP · Example 3 | deterministic global utility-optimal policy existence | Hurt |
| use a strictly increasing, continuously differentiable utility function | finite MOMDP · Example 4 | stochastic global utility-optimal policy existence | Hurt |
| use a discontinuous strictly increasing utility with u(x,y)=0 when x=y and 1/|x−y| otherwise | finite MOMDP · Example 5, state s1 | state-level utility-optimal policy existence | Hurt |
| require preference relations to be complete and transitive, with a maximal element at every state | finite MOMDP · preference representation | quasi-representative utility function existence | Helped |
Surprising findings
- Strictly monotonically increasing and continuously differentiable utility functions need not have a stochastic u-optimal policy (Example 4).
- A finite MOMDP with strictly monotonically increasing utility can have no utility-optimal policy at any state (Example 5), undermining the common state-independent criterion in MORL.
Limitations
- Purely theoretical; no experiments.
- Results give only sufficient conditions for existence of optimal policies.
- Restricted to finite MOMDPs, single-agent setting, and SER criterion.
Experimental setup
Builds on: Markov decision process undominated set convex hull solution concept Skalse et al. objectives and linear utility representation
Closest papers by meaning
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.90)
- Multi-objective Linear Reinforcement Learning with Lexicographic Rewards (ICML 2025 · similarity 0.90)
- Preference Optimization on Pareto Sets: On a Theory of Multi-Objective Optimization (NeurIPS 2025 · similarity 0.89)
- Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.89)
- Towards Theoretical Understanding of Sequential Decision Making with Preference Feedback (ICML 2025 · similarity 0.89)
- The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.