Research map / Reinforcement learning / Visual and representation RL
Going Beyond Heuristics by Imposing Policy Improvement as a Constraint
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HEPO, a constrained policy optimization method that trains a policy with task and heuristic rewards under the constraint that its task return is at least that of a heuristic-only policy, implemented on PPO.
Problem
Heuristic rewards speed up RL but bias policies and cause reward hacking; existing policy-invariance methods (PBRS) guarantee unbiasedness but do not improve task performance and need careful tuning of the task/heuristic trade-off.
Approach
Replace policy invariance with a policy-improvement constraint J(pi) >= J(pi_H) over the heuristic-trained policy, solved via Lagrangian min-max; the multiplier alpha reweights task reward adaptively, and both policies are trained concurrently on shared trajectories with PPO.
Key results
- HEPO achieves IQM of normalized return above H-only across 29 IsaacGym and Bi-DexHands tasks with a 62% average probability of improvement over the heuristic policy.
- With non-expert designed heuristics on FrankaCabinet, HEPO gets IQM 0.94 vs 0.44 for H-only and a probability of improvement of 0.73.
- HEPO outperforms PBRS, HuRL, EIPO and J+H on the benchmark; PBRS yields no gain over J-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HEPO instead of H-only with benchmark heuristic rewards | PPO on Isaac and Bi-Dex tasks · Isaac + Bi-Dex, 29 tasks | probability of improvement over H-only: 62% | Helped |
| Use HEPO instead of J-only | PPO on Isaac and Bi-Dex tasks · Isaac + Bi-Dex, 29 tasks | IQM normalized return | Helped |
| Use HEPO instead of J+H | PPO on Isaac and Bi-Dex tasks · Isaac + Bi-Dex, 29 tasks | IQM normalized return | Helped |
| Use HEPO instead of PBRS | PPO on Isaac and Bi-Dex tasks · Isaac + Bi-Dex, 29 tasks | task return | Helped |
| Use HEPO instead of EIPO | PPO on Isaac and Bi-Dex tasks · Isaac + Bi-Dex, 29 tasks | task return | Helped |
| Use HEPO instead of HuRL | PPO on Isaac and Bi-Dex tasks · Isaac + Bi-Dex, 29 tasks | task return | Helped |
| Use HEPO rather than H-only with non-expert-designed heuristics | PPO on FrankaCabinet · FrankaCabinet, 12 human-designed heuristics | IQM normalized return: 0.44 (0.37, 0.52) -> 0.94 (0.85, 1.03) | Helped |
| Use HEPO rather than H-only with non-expert-designed heuristics | PPO on FrankaCabinet · FrankaCabinet, 12 human-designed heuristics | probability of improvement over H-only: 0.50 (0.49, 0.51) -> 0.73 (0.73, 0.74) | Helped |
| Set the constraint reference policy to J-only rather than H-only (ablation) | HEPO · Ant, FrankaCabinet, and AllegroHand | task return | Hurt |
| Collect trajectories with an alternating policy rather than jointly from both policies (ablation) | HEPO · Ant, FrankaCabinet, and AllegroHand | task return | Hurt |
| Vary the heuristic-reward coefficient across a wide range (ablation) | HEPO · Ant, FrankaCabinet, and AllegroHand | task return | No clear effect |
| Vary the initial value of the Lagrangian multiplier (ablation) | HEPO · Ant, FrankaCabinet, and AllegroHand | task return | No clear effect |
Surprising findings
- Potential-based reward shaping rarely improves over task-only training in practice, showing policy invariance guarantees do not hold in practice.
- Heuristic-only PPO with non-expert reward functions sometimes drops below the expert-heuristic baseline, while HEPO stays near or above it.
Limitations
- No theoretical convergence guarantee to the optimal policy.
- Requires training two policies, which may hinder use with large language models due to memory demands.
- Hyperparameter sensitivity was tested only on three tasks.
Experimental setup
Datasets: IsaacGym tasks (9) Bi-DexHands tasks (20)
Benchmarks: IsaacGym Bi-DexHands
Metrics: interquartile mean of normalized return probability of improvement task return
Baselines: H-only J-only J+H PBRS HuRL EIPO
Builds on: PPO EIPO potential-based reward shaping HuRL generalized advantage estimation
Closest papers by meaning
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.88)
- Hindsight PRIORs for Reward Learning from Human Preferences (ICLR 2024 · similarity 0.88)
- Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.88)
- To the Max: Reinventing Reward in Reinforcement Learning (ICML 2024 · similarity 0.88)
- Learning Process Rewards via Visitation Matching for Efficient RL (NeurIPS 2026 · similarity 0.87)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.