Research map / Reinforcement learning / Imitation learning

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning

NeurIPS 2024 · Reinforcement learning › Imitation learning · approach: Theory of inverse RL · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies when rewards recovered by regularized IRL from finite expert demonstrations transfer to new transition laws, introducing principal angles between transition laws and giving sufficient conditions plus a PAC multi-expert IRL algorithm.

Problem

Rewards recovered by IRL from finite expert demonstrations may not have optimal policies that align with the expert's true reward under new transition laws; prior transferability guarantees assumed full access to expert policies.

Approach

Replaces the binary rank condition with principal angles between transition-law subspaces as a refined similarity measure; shows second principal angle across experts gives universal transferability and maximal principal angle gives local transferability, under steep, strongly convex regularization.

Key results

• Example shows that reward recovered from two approximately optimal experts can have suboptimality ~4.81 in a new environment despite satisfying the rank condition, so full-access conditions fail with finite data.
• Theorem gives transferability to all transition laws with epsilon = eps_hat / (eta * sigma_R * sin(theta_2/2)^2) when learning from two experts with nonzero second principal angle.
• Gridworld experiments show closeness of recovered reward and transferability improve as second principal angle increases (wind strength grows).

Measured effects reported in the paper

Surprising findings

• Conditions developed under full access to expert policies (rank condition) are insufficient for transferability when learning from finite demonstrations.

Limitations

• Results are only sufficient conditions; necessary conditions and tightness are unknown.
• Restricted to discrete state and action spaces and full-dimensional reward class; constants in bounds are large.
• Experiments are limited to a gridworld and no real-world application.

Experimental setup

Datasets: Stochastic WindyGridworld (synthetic expert demonstrations)

Benchmarks: WindyGridworld

Metrics: distance of recovered reward to true reward in quotient spac transferability suboptimality on South-wind and shifted tran second principal angle

Baselines: Rank-condition-based transferability results (Cohen et al. 2021 style)

Builds on: Relative entropy IRL / regularized IRL (Cohen et al. style steep regularization potential shaping equivalence Bregman divergence duality

Closest papers by meaning

• Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.92)
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.91)
• Sub-optimal Experts mitigate Ambiguity in Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
• Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.89)
• Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.89)
• Inverse Reinforcement Learning by Estimating Expertise of Demonstrators (AAAI 2025 · similarity 0.88)