Research map / Reinforcement learning / Visual and representation RL
Zero-Shot Reinforcement Learning from Low Quality Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows zero-shot RL methods (forward-backward representations) overestimate OOD action values on small low-quality datasets and proposes conservative regularisers (VC-FB, MC-FB) that match or beat FB and even task-aware CQL on ExORL.
Problem
Zero-shot RL methods based on successor measures and successor features need large heterogeneous pre-training datasets; their performance collapses when pre-trained on small, low-quality, homogeneous offline datasets due to OOD value overestimation.
Approach
Re-purpose CQL-style conservatism for zero-shot RL by penalising OOD action values (VC-FB) or OOD successor measures (MC-FB) across task vectors sampled from the task distribution, without introducing new parametric functions, with the penalty weight tuned via Lagrangian dual gradient descent.</ېcore_idea>
Key results
- On ExORL 100k datasets, MC-FB and VC-FB reach 150% and 137% of FB's IQM, and VC-FB/MC-FB reach 120%/111% of single-task CQL's IQM.
- On Rnd 100k, VC-FB and MC-FB reach 2.5x and 1.8x of FB performance.
- On D4RL, FB reaches only 10% of CQL's aggregate performance, while MC-FB and VC-FB improve on FB by 5.6x and 6.8x but still underperform CQL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add value-conservative regularization | FB · ExORL all datasets, domains and tasks (100k transitions) | Aggregated performance: 99 -> 148 | Helped |
| Add measure-conservative regularization | FB · ExORL all datasets, domains and tasks (100k transitions) | Aggregated performance: 99 -> 136 | Helped |
| Use backward-model embeddings for all conservative-update task vectors (ablation) | VC-FB · ExORL all datasets, domains and tasks (100k transitions) | Aggregated performance: 148 -> 108 | Hurt |
| Add value-conservative regularization | FB · ExORL all datasets, domains and tasks (full datasets) | IQM score: 256 -> 267 | Helped |
| Add measure-conservative regularization | FB · ExORL all datasets, domains and tasks (full datasets) | IQM score: 256 -> 271 | Helped |
| Add value-conservative regularization | FB · ExORL Random, all domains and tasks (full dataset) | IQM score: 111 -> 131 | Helped |
| Add measure-conservative regularization | FB · ExORL Random, all domains and tasks (full dataset) | IQM score: 111 -> 133 | Helped |
| Add value-conservative regularization | FB · ExORL aggregate across datasets, domains and tasks | IQM performance relative to FB: 150% | Helped |
| Add measure-conservative regularization | FB · ExORL aggregate across datasets, domains and tasks | IQM performance relative to FB: 150% | Helped |
| Add value-conservative regularization | FB · D4RL aggregate across domains and datasets | Performance relative to FB: 6.8 times | Helped |
| Add measure-conservative regularization | FB · D4RL aggregate across domains and datasets | Performance relative to FB: 5.6 times | Helped |
| Increase the Rnd dataset size | FB, VC-FB and MC-FB · ExORL aggregate across domains | IQM performance gap between conservative variants and FB | Hurt |
Surprising findings
- Conservative zero-shot variants outperform single-task CQL, which is trained with task reward labels, on ExORL 100k datasets.
- Conservative variants do not outperform CQL on D4RL, attributed to needing careful alpha selection.
Limitations
- Dataset quality is proxied by offline TD3 performance on Maze, an imperfect metric.
- Conservative variants underperform CQL on D4RL, requiring careful alpha tuning.
- Avoiding new parametric functions limits the design space; methods like IQL or AWAC are not explored.
Experimental setup
Datasets: ExORL D4RL
Benchmarks: ExORL D4RL
Metrics: IQM of task score normalised score vs CQL performance profiles
Baselines: FB SF-LAP GC-IQL CQL offline TD3 DVC-FB
Builds on: forward-backward representations universal successor features conservative q-learning ExORL D4RL
Closest papers by meaning
- Provable Zero-Shot Generalization in Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
- Zero-Shot Reinforcement Learning via Function Encoders (ICML 2024 · similarity 0.90)
- Towards Robust Zero-Shot Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Zero Shot Generalization of Vision-Based RL Without Data Augmentation (ICML 2025 · similarity 0.89)
- Proto Successor Measure: Representing the Behavior Space of an RL Agent (ICML 2025 · similarity 0.89)
- Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.