Research map / Reinforcement learning / MDPs and planning

Learning Utilities from Demonstrations in Markov Decision Processes

ICML 2025 · Reinforcement learning › MDPs and planning · approach: POMDPs and confounded RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a risk-sensitive model of expert behavior in MDPs (expert optimal in an RS-MDP with learnable utility over returns), analyzes partial identifiability of utilities, and gives two provably-efficient algorithms (CATY, TRACTOR) for utility learning from finite demonstrations.

Problem

Standard IRL assumes risk-neutral experts and Markovian policies, causing misspecification and not allowing direct inference of an agent's risk attitude; the utility function of a risk-sensitive expert is only partially identifiable from demonstrations.

Approach

Model the expert as an optimal agent in a risk-sensitive MDP whose utility is applied to the full return, so optimal policies are non-Markovian (depending on cumulative reward); learn the utility (the risk attitude) separately from the reward, using demonstrations from multiple environments to reduce non-identifiability, via enlarged-state-space planning and discretization.

Key results

• 10 of 15 participants exhibited non-Markovian policies in the H=5 experiment, supporting the new model.
• A Markovian-vs-non-Markovian gap of 0.5 is shown to exist in a constructed RS-MDP (Proposition on performance loss).
• CATY correctly classifies utilities inside U_{Delta-eps} or outside U_{Delta+eps} with sample complexity independent of Delta; TRACTOR converges with T >= O(N^4H^4L^2/eps^4) gradient iterations.

Measured effects reported in the paper

Surprising findings

• Demonstrations in a single environment leave utilities almost unconstrained: two feasible utilities can differ by 1 in the all-policies distance.
• Non-Markovian optimal policies can outperform every Markovian policy by a gap of 0.5 in constructed examples.

Limitations

• Only utility learning with known reward and dynamics is addressed; jointly learning reward and utility is left to future work.
• Partial identifiability persists with single-environment demonstrations.
• Experiments are small proof-of-concept with 15 participants and small horizon.

Experimental setup

Datasets: human demonstrations from 15 lab participants (H=5 MDP) simulated MDPs

Metrics: fraction of participants exhibiting non-Markovian behavior number of gradient iterations to reach (non)compatibility le compatibility cost

Builds on: enlarged state space approach for RS-MDPs inverse constrained reinforcement learning categorical return distribution representation maximum entropy IRL

Closest papers by meaning

• Teaching Inverse Reinforcement Learners via Features and Demonstrations (NeurIPS 2018 · similarity 0.90)
• Inverse Decision Modeling: Learning Interpretable Representations of Behavior (ICML 2021 · similarity 0.89)
• Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting (AAAI 2025 · similarity 0.89)
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.88)
• Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification (ICLR 2024 · similarity 0.88)
• Environment Design for Inverse Reinforcement Learning (ICML 2024 · similarity 0.88)