Research map / Reinforcement learning / Imitation learning
Reward-Estimated Hypergradient for Bilevel Reinforcement Learning with Black-Box Follower
Shigeki Kusaka, Mikoto Kudo, Takumi Tanabe, Akifumi Wachi, Youhei Akimoto
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reward-Estimated Hypergradients (RE-HG), which use inverse RL to estimate the black-box follower's reward from observed trajectories so the leader can compute hypergradients in bilevel RL.
Problem
Bilevel reinforcement learning with hypergradient-based leader optimization typically assumes a white-box follower whose reward is known; this fails in real-world settings where the leader only observes the follower's trajectories.
Approach
Estimates the follower's unknown reward function via inverse RL from observed trajectories and plugs the estimate into the hypergradient computation, so the leader can optimize without access to the follower's true reward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the influence term to the leader hypergradient (ablation) | RE-HG / Oracle · Discrete 2-player Markov game toy task | leader discounted return | Improved |
| aggregate IRL estimates across outer iterations with eigenvalue truncation instead of a single IRL at the start (ablation) | RE-HG · Building thermal control (configurable MDP) | leader discounted return / gradient norm: gradient norm about 10^7 -> about 10^1 | Improved |
| aggregate IRL estimates across outer iterations (ablation) | RE-HG · Building thermal control (configurable MDP) | HG cosine similarity and reward Pearson correlation | Improved |
| use estimated smoothed reward (RE-HG) instead of exact Oracle hypergradient | Oracle · Building thermal control (steep reward landscape) | leader discounted return | Improved |
| Oracle with tuned learning rate, gradient clipping, or scaled influence term (ablation) | Oracle · Building thermal control | leader discounted return | No clear effect |
| use RE-HG with recovered reward instead of white-box Oracle | RE-HG · Continuous 2-player Markov game (LQR) | leader return | No clear effect |
Experimental setup
Builds on: hypergradient methods for bilevel RL inverse reinforcement learning
Closest papers by meaning
- Contextual Bilevel Reinforcement Learning for Incentive Alignment (NeurIPS 2024 · similarity 0.91)
- PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.87)
- A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Towards Global Optimality for Practical Average Reward Reinforcement Learning without Mixing Time Oracles (ICML 2024 · similarity 0.86)
- Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.86)
- Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
