Research map / Reinforcement learning / Imitation learning

Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting

AAAI 2025 · Reinforcement learning › Imitation learning · approach: Theory of inverse RL · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Characterises partial identifiability of the reward function in IRL for agents with non-exponential (e.g. hyperbolic) discounting, by defining resolute, naive and sophisticated policies and corresponding Boltzmann behavioural models, and proving they determine reward only up to potential shaping (naive/sophisticated) and are generally too ambiguous to transfer across discount functions.

Problem

IRL reward identifiability has only been studied for exponentially discounting agents, yet humans are better modelled as hyperbolic discounters, whose preferences can be temporally inconsistent and whose optimal policy is ill-defined.

Approach

Generalises Boltzmann-rationality to general discount functions via three notions of policy (resolute, naive, sophisticated), defines three Boltzmann behavioural models, and proves exact and qualitative partial-identifiability results (reward determined up to potential shaping for naive/sophisticated models; general ambiguity across discount functions).

Key results

• Boltzmann-naive and Boltzmann-sophisticated behavioural models determine the reward up to naive and sophisticated potential shaping respectively, for any episodic MDP and discount function.
• For any discount function other than exponential (up to horizon |S|-1), there exist episodic transition functions where the reward is too ambiguous to recover the optimal policy under a different discount function.
• Resolute, naive and sophisticated optimal policies always exist in any episodic MDP, while there are episodic MDPs with no stationary resolute policy.

Measured effects reported in the paper

Surprising findings

• IRL alone is generally insufficient to identify the correct optimal policy for agents with non-exponential discounting, even with infinite data.
• The set of sophisticated policies is not convex, and there need not be a unique sophisticated Q-function in a given MDP.

Limitations

• Analysis restricted to episodic MDPs with bounded horizon in the main text; non-episodic MDPs are not covered.
• Boltzmann-resolute model's partial identifiability is not exactly characterised.
• Does not assess which behavioural model best fits human data; purely mathematical with no empirical validation.

Experimental setup

Builds on: Boltzmann rationality potential shaping Ng et al. reward identifiability framework

Closest papers by meaning

• Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification (ICLR 2024 · similarity 0.91)
• Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.90)
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.89)
• Learning Utilities from Demonstrations in Markov Decision Processes (ICML 2025 · similarity 0.89)
• Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.88)
• When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback (NeurIPS 2024 · similarity 0.88)