AI Paper Atlas

Research map / Reinforcement learning / Imitation learning

Reward-Estimated Hypergradient for Bilevel Reinforcement Learning with Black-Box Follower

Shigeki Kusaka, Mikoto Kudo, Takumi Tanabe, Akifumi Wachi, Youhei Akimoto

NeurIPS 2026 · Reinforcement learning › Imitation learning · approach: Imitation combined with RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Reward-Estimated Hypergradients (RE-HG), which use inverse RL to estimate the black-box follower's reward from observed trajectories so the leader can compute hypergradients in bilevel RL.

Problem

Bilevel reinforcement learning with hypergradient-based leader optimization typically assumes a white-box follower whose reward is known; this fails in real-world settings where the leader only observes the follower's trajectories.

Approach

Estimates the follower's unknown reward function via inverse RL from observed trajectories and plugs the estimate into the hypergradient computation, so the leader can optimize without access to the follower's true reward.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add the influence term to the leader hypergradient (ablation)RE-HG / Oracle · Discrete 2-player Markov game toy taskleader discounted returnImproved
aggregate IRL estimates across outer iterations with eigenvalue truncation instead of a single IRL at the start (ablation)RE-HG · Building thermal control (configurable MDP)leader discounted return / gradient norm: gradient norm about 10^7 -> about 10^1Improved
aggregate IRL estimates across outer iterations (ablation)RE-HG · Building thermal control (configurable MDP)HG cosine similarity and reward Pearson correlationImproved
use estimated smoothed reward (RE-HG) instead of exact Oracle hypergradientOracle · Building thermal control (steep reward landscape)leader discounted returnImproved
Oracle with tuned learning rate, gradient clipping, or scaled influence term (ablation)Oracle · Building thermal controlleader discounted returnNo clear effect
use RE-HG with recovered reward instead of white-box OracleRE-HG · Continuous 2-player Markov game (LQR)leader returnNo clear effect

Experimental setup

Builds on: hypergradient methods for bilevel RL inverse reinforcement learning

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy