Research map / Reinforcement learning / Visual and representation RL
Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Functional Reward Encodings (FRE), a transformer-based VAE that encodes sampled state-reward sets into a latent task vector, pretrained on random unsupervised reward functions to enable zero-shot offline RL on new tasks.
Problem
Zero-shot reinforcement learning from unlabeled offline trajectories: pretrain a generalist agent that can solve arbitrary new reward functions at test time without further training, given only a few reward-annotated samples.
Approach
Learn a latent encoding of reward functions directly from sampled (state, reward) pairs using a permutation-invariant transformer VAE with an information-bottleneck objective, trained over a mixture of random reward functions (goal-reaching, random linear, random MLP); then train an IQL policy conditioned on the latent, so new tasks are solved by encoding a few reward samples.
Key results
- FRE achieves 57 ± 9 average normalized score across all tasks versus 24 for FB, 18 for SF, and 33 for OPAL-10.
- On AntMaze, FRE scores 52.8 ± 18.2 overall vs 25.8 for FB, using only 32 reward samples at test time versus 5120 for FB/SF.
- On Kitchen, FRE scores 66 ± 3 vs 3 for FB and 59 for GC-IQL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FRE rather than FB | zero-shot offline RL · AntMaze all | normalized score: 25.8 ± 19.8 -> 52.8 ± 18.2 | Helped |
| use FRE rather than FB | zero-shot offline RL · ExORL all | normalized score: 43.4 ± 9.1 -> 51.5 ± 6.3 | Helped |
| use FRE rather than GC-IQL | zero-shot offline RL · Kitchen | normalized score: 59 ± 4 -> 66 ± 3 | Helped |
| use FRE rather than FB | zero-shot offline RL · All benchmarks | normalized score: 24 ± 12 -> 57 ± 9 | Helped |
| add goal-reaching and linear rewards to MLP rewards (ablation) | FRE · AntMaze total | normalized score: 25.3 ± 8 -> 47.3 ± 7 | Helped |
| add linear and MLP rewards to goal-reaching rewards (ablation) | FRE · AntMaze total | normalized score: 26.1 ± 8 -> 47.3 ± 7 | Helped |
| add goal-reaching and MLP rewards to linear rewards (ablation) | FRE · AntMaze total | normalized score: 31.6 ± 5 -> 47.3 ± 7 | Helped |
| add goal-reaching rewards to linear and MLP rewards (ablation) | FRE · AntMaze total | normalized score: 32.3 ± 5 -> 47.3 ± 7 | Helped |
| add linear rewards to goal-reaching and MLP rewards (ablation) | FRE · AntMaze total | normalized score: 33.8 ± 15 -> 47.3 ± 7 | Helped |
| add MLP rewards to goal-reaching and linear rewards (ablation) | FRE · AntMaze total | normalized score: 46.9 ± 7 -> 47.3 ± 7 | No clear effect |
Surprising findings
- FB and SF, despite strong zero-shot claims, perform poorly on goal-reaching in AntMaze (0.0 and 0.4) and Kitchen compared with FRE.
Limitations
- Relies on a hand-specified prior reward distribution that is somewhat ad hoc.
- Formulation requires offline data; extension to online setting left to future work.
- Some trajectories fail on out-of-distribution states, a common offline RL challenge.
Experimental setup
Datasets: D4RL antmaze-large-diverse-v2 ExORL walker ExORL cheetah D4RL kitchen
Benchmarks: D4RL AntMaze ExORL D4RL Kitchen
Metrics: normalized return (0-100)
Baselines: Forward-Backward (FB) Successor Features (SF) GC-IQL GC-BC OPAL
Builds on: implicit Q-learning forward-backward representations successor features neural processes hindsight experience replay
Closest papers by meaning
- Zero-Shot Reinforcement Learning via Function Encoders (ICML 2024 · similarity 0.93)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.90)
- Zero-Shot Reinforcement Learning from Low Quality Data (NeurIPS 2024 · similarity 0.88)
- Skill-aware Mutual Information Optimisation for Zero-shot Generalisation in Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Towards Robust Zero-Shot Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.