Research map / Reinforcement learning / Visual and representation RL
Zero-Shot Reinforcement Learning via Function Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the function encoder, a learned linear-basis representation of functions (rewards or transitions) that is fed into standard RL algorithms to enable zero-shot transfer across related tasks.
Problem
Zero-shot transfer across a continuum of related RL tasks that differ in reward or transition functions requires a task representation that captures how a new task relates to previously seen tasks.
Approach
Represent each perturbing function (reward, transition, or adversary policy) as a coefficient vector over learned non-linear basis functions, computed via Monte-Carlo inner products on example data; the linear coefficient vector is passed as an extra input to the policy and value functions of any RL algorithm.
Key results
- On hidden-parameter Half-Cheetah, FE + MLP achieves a 37.5% decrease in test MSE relative to the transformer baseline; FE alone gives 19.7%.
- On multi-task Ms. Pacman, DQN + FE achieves about a 20% higher success rate than multi-task baselines and better data efficiency than DQN + Trans.
- On the partially observable game of tag, PPO + FE significantly outperforms PPO and PPO + Trans and matches the PPO + OHE oracle.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Represent transition functions with a function encoder instead of a transformer | Half-Cheetah system identification · Half-Cheetah | test MSE: 19.7% decrease in MSE | Helped |
| Represent the average transition function with an MLP and the difference with a function encoder | Half-Cheetah system identification · Half-Cheetah | test MSE: 37.5% decrease in MSE | Helped |
| Use an MLP without example data to predict transitions | Half-Cheetah system identification · Half-Cheetah | test MSE | Hurt |
| Add function-encoder representations of adversary policies | PPO · Partially observable tag | return | Helped |
| Provide a one-hot encoding of the current adversary | PPO · Partially observable tag | return | Helped |
| Use adversary data directly as transformer input instead of a principled representation | PPO · Partially observable tag | return | Hurt |
| Add reward-function representations from a function encoder | DQN · Ms. Pacman | success rate: 20% higher success rate | Helped |
| Replace basis-function reward reconstruction with an MLP conditioned on the representation (ablation) | DQN + FE · Ms. Pacman | success rate | Hurt |
Surprising findings
- The transformer baseline needs about 194% more training time than an MLP, while the function encoder adds only about 5%.
Limitations
- Assumes example data on the perturbing function is available at execution; does not address exploration to collect this data.
- Zero-shot guarantee only holds for unseen tasks that are linear combinations of the learned basis functions.
- Orthonormality is not enforced in training and is only observed empirically.
Experimental setup
Datasets: Half-Cheetah hidden-parameter environment Game of tag (pre-trained adversary league) Multi-task Ms. Pacman
Benchmarks: Half-Cheetah hidden-parameter system identification Partially observable game of tag Multi-task Ms. Pacman
Metrics: test MSE return success rate distance to objective cosine similarity of representations
Baselines: MLP Transformer Oracle PPO PPO + OHE PPO + Trans FB DQN + HER DQN + Trans DQN + FE (abl.)
Builds on: PPO DQN Hindsight Experience Replay forward-backward representation learning
Closest papers by meaning
- Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings (ICML 2024 · similarity 0.93)
- Zero-Shot Reinforcement Learning from Low Quality Data (NeurIPS 2024 · similarity 0.90)
- Towards Robust Zero-Shot Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Proto Successor Measure: Representing the Behavior Space of an RL Agent (ICML 2025 · similarity 0.88)
- Skill-aware Mutual Information Optimisation for Zero-shot Generalisation in Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Zero-Shot Instruction Following in RL via Structured LTL Representations (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.