Research map / Reinforcement learning / Offline RL
Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a model-free Eikonal PDE gradient-norm regularizer to implicit value learning in offline goal-conditioned RL, yielding Eik-HIQL, which improves HIQL on OGBench navigation and stitching tasks.
Problem
Accurate goal-conditioned value function (GCVF) estimation from limited-coverage offline data for long-horizon goal-reaching tasks, with poor generalization to unseen state-goal pairs.
Approach
Adds a physics-informed regularizer derived from the Eikonal PDE, (||grad_s V|| * S(s) - 1)^2 with S(s)=1, to the expectile value loss of HIQL, imposing a distance-like cost-to-go structure without requiring known dynamics.
Key results
- Eik-HIQL achieves 22 ± 10 on pointmaze-giant-stitch vs 0 ± 0 for HIQL-style HJB variant and improves over HIQL on large and giant mazes, with gains exceeding 100% in 7 of 31 tasks.
- Eik-HIQL matches or outperforms QRL, CRL and HIQL on navigation and stitching tasks, with limited or no gain on antsoccer and manipulation tasks.
- Eikonal regularizer integrated with GCIVL yields e.g. 86 ± 11 vs 0 ± 0 on pointmaze-giant-navigate, showing compatibility with other TD-based GCRL methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the constant speed profile S(s)=1 instead of the exponential or linear speed profile (ablation) | Eik-HIQL · pointmaze navigate and stitch, medium/large/giant/teleport | success rate: 83 ± 9 vs 61 ± 8 and 60 ± 5; 79 ± 13 vs 38 ± 15 and 42 ± 12; 73 ± 6 vs 33 ± 17 and 41 ± 7; 22 ± 10 vs 5 ± 8 and 2 ± 3 | Helped |
| Use the constant speed profile S(s)=1 instead of the exponential or linear speed profile (ablation) | Eik-HIQL · pointmaze navigate medium and stitch teleport | success rate: 94 ± 4 vs 95 ± 2 and 93 ± 4; 44 ± 9 vs 43 ± 5 and 38 ± 7 | Mixed |
| Replace the Eikonal regularizer with the HJB regularizer (ablation) | HIQL · pointmaze navigate and stitch | success rate: 94 ± 4 vs 90 ± 6; 83 ± 9 vs 53 ± 9; 79 ± 13 vs 9 ± 8; 47 ± 10 vs 18 ± 5; 97 ± 2 vs 64 ± 13; 73 ± 6 vs 6 ± 7; 22 ± 10 vs | Hurt |
| Add the Eikonal regularizer to HIQL | HIQL · navigation tasks, especially large mazes and stitching regimes | performance | Helped |
Surprising findings
- A constant speed profile S(s)=1 outperformed obstacle-aware speed profiles that require obstacle coordinates.
- The HJB-style finite-difference regularizer performed worse than the Eikonal residual and failed to recover maze geometry.
Limitations
- Underperforms or matches baselines in contact-rich tasks (antsoccer, manipulation) where value functions are discontinuous.
- Eikonal regularizer induces excessive smoothness that hurts interactive tasks.
- Speed profile designs that encode obstacle information require privileged knowledge of obstacle coordinates and were not beneficial in experiments.
Experimental setup
Datasets: OGBench pointmaze OGBench antmaze OGBench humanoidmaze OGBench antsoccer OGBench cube OGBench scene
Benchmarks: OGBench pointmaze OGBench antmaze OGBench humanoidmaze OGBench antsoccer OGBench cube OGBench scene
Metrics: success rate (%)
Baselines: HIQL QRL CRL HJB-HIQL GCIQL GCIVL
Builds on: HIQL Implicit Q-Learning OGBench Hamilton-Jacobi-Bellman regularization
Closest papers by meaning
- Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Conservative Offline Goal-Conditioned Implicit V-Learning (ICML 2025 · similarity 0.90)
- OGBench: Benchmarking Offline Goal-Conditioned RL (ICLR 2025 · similarity 0.88)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.88)
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.