Research map / Reinforcement learning / Visual and representation RL
$\pi$2vec: Policy Representation with Successor Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces π2vec, which represents black-box policies as vectors by averaging offline-trained successor features over pretrained visual foundation model features, then regresses policy returns from these vectors to predict held-out policy performance.
Problem
Predicting the performance of held-out black-box policies without running them in the environment, which requires comparable policy representations learned from offline data.
Approach
Represent a policy by the expected discounted sum of pretrained visual features along its trajectories (successor features), trained offline with distributional FQE, then average over canonical states and regress returns linearly on these vectors, replacing action-based policy representations.
Key results
- On real RGB Stacking, the VIT-based pi2vec achieves Regret@1 0.036 versus 0.074 for the Actions baseline (51% relative improvement).
- On real Insert Gear, CLIP-based pi2vec reaches correlation 0.618 versus -0.545 for Actions and Regret@1 0.267 versus 0.578.
- Averaged over Metaworld, BEST-phi pi2vec attains Regret@1 0.153 versus 0.232 for Actions and correlation 0.460 versus 0.347.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| represent policies with successor features and pretrained state features instead of canonical-state actions | π2vec vs Actions · Insert gear (real) | Regret@1: 0.578 -> 0.267 | Helped |
| represent policies with VIT successor features instead of canonical-state actions | π2vec vs Actions · RGB Stacking | Regret@1: 0.074 -> 0.036 | Helped |
| select the best feature encoder per task-setting | π2vec · Metaworld | Regret@1: 0.232 -> 0.153 | Helped |
| select the best feature encoder per task-setting | π2vec · Kitchen | Regret@1: 0.221 -> 0.070 | Helped |
| replace state features φ(s) with transition differences Δφ(s,s') (ablation) | π2vec · RGB Stacking | Correlation: 0.775 -> 0.030 | Hurt |
| replace state features φ(s) with transition differences Δφ(s,s') (ablation) | π2vec · Metaworld | Regret@1: 0.263 -> 0.325 | Mixed |
| train successor features on trajectories rather than demonstrations (ablation) | π2vec on Metaworld · Metaworld | Regret@1: 0.232 -> 0.189 | Helped |
| use the best available dataset per task rather than demonstrations (ablation) | π2vec BEST-φ on Metaworld · Metaworld | Regret@1: 0.187 -> 0.135 | Helped |
| use first states of demonstrations as canonical states, limiting canonical-state coverage (ablation) | π2vec ΔCLIP · Assembly (left) | Regret@1: null | No clear effect |
Surprising findings
- Action-based representations generalize poorly across intermediate checkpoints of similar policies, yielding negative correlation (-0.189) while the pi2vec variant achieves positive correlation.
- A randomly initialized ResNet-50 feature encoder performs poorly on Metaworld and Kitchen, showing the choice of foundation-model encoder is critical.
Limitations
- Return prediction requires some policies with known performance (not fully offline), and NMAE is high in the fully-offline setting.
- Successor feature assumption of linear rewards is violated, so the method regresses returns directly rather than using the reward factorization.
- Choosing the best feature encoder per task requires cross-validation on evaluated policies, and no single encoder dominates across settings.
Experimental setup
Datasets: Metaworld demonstrations Franka-Kitchen human demonstrations Insert Gear (sim) trajectories Insert Gear (real) trajectories RGB Stacking offline trajectories
Benchmarks: Metaworld Franka-Kitchen Insert Gear (sim) Insert Gear (real) RGB Stacking
Metrics: NMAE rank correlation Regret@1
Baselines: Actions (AOPS policy representation) FQE Random ResNet-50 features
Builds on: successor features fitted Q evaluation active offline policy selection (AOPS) CLIP track-any-point
Closest papers by meaning
- Distributional Successor Features Enable Zero-Shot Policy Optimization (NeurIPS 2024 · similarity 0.86)
- Bridging State and History Representations: Understanding Self-Predictive RL (ICLR 2024 · similarity 0.84)
- Learning Successor Features the Simple Way (NeurIPS 2024 · similarity 0.84)
- Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees (ICLR 2024 · similarity 0.83)
- A Distributional Analogue to the Successor Representation (ICML 2024 · similarity 0.83)
- Reward-Aware Proto-Representations in Reinforcement Learning (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.