Research map / Reinforcement learning / Visual and representation RL
Foundation Policies with Hilbert Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hilbert foundation policies (HILPs), which pre-train a latent-conditioned policy from unlabeled offline data by learning a temporal-distance-preserving Hilbert representation and spanning it with directional inner-product rewards, enabling zero-shot RL, goal-reaching, and hierarchical RL.
Problem
Finding a general, scalable unsupervised pre-training objective for generalist RL policies from unlabeled offline data, where existing approaches (behavioral cloning, offline GCRL, skill discovery, successor features) are limited in behavior diversity, data quality requirements, or adaptation mechan
Approach
Learn a Hilbert representation phi whose L2 distances match optimal temporal distances (via goal-conditioned value learning with V = -||phi(s)-phi(g)||), then train a latent-conditioned policy with offline RL on the inner-product reward <phi(s')-phi(s), z> over unit vectors z, so the policy spans the latent space; the representation then enables zero-shot prompting (linear regression on reward for
Key results
- On state-based ExORL zero-shot RL, HILP achieves the best overall IQM across 4 environments, 4 tasks, 4 datasets, and 4 seeds, outperforming FB and SF baselines.
- On goal-conditioned D4RL tasks, HILP-Plan with test-time midpoint planning reaches 64.5 on antmaze-large-diverse and 59.2 on antmaze-ultra-diverse, often matching or exceeding GC-IQL.
- In hierarchical RL, HILP averages 51.3 versus 45.7 for OPAL, with 43.8 vs 21.0 on antmaze-ultra-diverse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add test-time midpoint planning (three recursions) (ablation) | HILP · AntMaze and Kitchen goal-conditioned tasks | performance | Mixed |
| increase midpoint-planning recursions from 0 to 3 (ablation) | HILP · offline goal-conditioned RL | performance | Helped |
| use latent dimension D=50 rather than other tested dimensions (ablation) | HILP · ExORL zero-shot RL | performance | Helped |
| vary latent dimension, using a dimension between 25 and 64 (ablation) | HILP · zero-shot RL and offline goal-conditioned RL | performance | Helped |
| use the centered representation reward instead of adjacent-state displacement reward (ablation) | HILP · zero-shot RL | performance | Helped |
| use HILP rather than OPAL for hierarchical skill learning | high-level IQL policy using offline skills · AntMaze and Kitchen hierarchical RL | performance: 45.7 -> 51.3 | Mixed |
Surprising findings
- Test-time midpoint planning with pre-computed representations often lets HILP outperform dedicated offline GCRL methods such as GC-IQL.
- HILP's zero-shot prompting underperforms OPAL on antmaze-large-diverse and kitchen-mixed in hierarchical RL despite the higher average.
Limitations
- Symmetric Hilbert space may not capture asymmetric or disconnected MDPs such as those where gravity matters.
- Value-based representation objective may be optimistically biased in stochastic or partially observable environments.
- Assumes the same state space and dynamics at evaluation time; no multi-environment or transfer evaluation.
Experimental setup
Datasets: ExORL D4RL antmaze-large D4RL antmaze-ultra D4RL kitchen visual-kitchen
Benchmarks: ExORL pixel-based ExORL D4RL antmaze-large D4RL antmaze-ultra D4RL kitchen visual-kitchen
Metrics: interquartile mean normalized return normalized return (goal-reaching success) average return
Baselines: FB FDM Lap AE IDM CL Rand GC-TD3 GC-BC GC-IQL GC-CQL OPAL
Builds on: IQL TD3 METRA successor features hindsight relabeling
Closest papers by meaning
- Representation Matters: Offline Pretraining for Sequential Decision Making (ICML 2021 · similarity 0.91)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.90)
- METRA: Scalable Unsupervised RL with Metric-Aware Abstraction (ICLR 2024 · similarity 0.90)
- Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings (ICML 2024 · similarity 0.90)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.89)
- RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.