Research map / Reinforcement learning / Imitation learning

A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Imitation learning · approach: Reward learning from demonstrations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Gives an occupancy-space (e-geodesic) view of reward updates in entropy-regularized reverse-KL IRL, showing the integral-FIM-preconditioned dual gradient equals a convex interpolation of old and target rewards, which is mirror descent and induces approximate entropic MD on policies, with sublinear convergence.

Problem

Reward parameter updates in primal-dual (Lagrangian) inverse reinforcement learning ignore the geometry of the occupancy space, where the optimal solution is explicitly known, leading to poor gradient-based reward updates.

Approach

Views IRL reward updates as moving the agent occupancy toward a known reward-dependent target occupancy along an e-geodesic; the resulting update preconditions the dual gradient by the average Fisher information along that path, which reduces to a convex interpolation of the current reward and the log density-ratio target, requiring no explicit FIM computation.

Key results

• Functional (FIM-preconditioned) updates align better with the direct optimisation direction than vanilla gradients at iteration 0 (32.1 vs 80.2 degrees) and at most later iterations, with natural gradient comparable.
• Contraction coefficient with norm-change correction is much better for functional updates than vanilla gradient (e.g., 0.7036 vs 0.9771 at iteration 10).
• Sublinear convergence of the saddle-point residual up to a fixed residual level B^2 L_F L_G/(4 beta m) + O(1/sqrt(T)).

Surprising findings

• Functional and natural gradients are often comparable in alignment, and both significantly outperform vanilla gradients in the gridworld experiment.
• The FIM-preconditioned direction is not always better than the natural gradient; natural gradient aligned better at most iterations after the first.

Limitations

• Does not account for function approximation errors, data availability, or density-ratio estimation errors.
• Assumes occupancies and rewards are linear in a finite feature vector; feature learning introduces approximation error.
• Reward-player regret bound relies on an assumption (bounded reward domain / Lipschitz conditions) that may not hold in practice.

Experimental setup

Benchmarks: 25-state gridworld

Metrics: angle deviation to direct optimisation direction contraction coefficient condition number of preconditioned curvature gradient error under feature misspecification

Baselines: Vanilla Gradient Natural Gradient TRIRL

Builds on: MaxCausalEnt-IRL Trust Region Inverse RL (TRIRL) natural gradient entropic mirror descent

Closest papers by meaning

• Recent Advancements in Inverse Reinforcement Learning (AAAI 2024 · similarity 0.91)
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.90)
• How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach (NeurIPS 2024 · similarity 0.90)
• Occupancy-based Policy Gradient: Estimation, Convergence, and Optimality (NeurIPS 2024 · similarity 0.89)
• Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.89)
• In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before an Ongoing Trajectory Terminates (NeurIPS 2024 · similarity 0.89)