Research map / Reinforcement learning / Offline RL
Towards Robust Zero-Shot Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BREEZE, a behavior-regularized forward-backward (FB) zero-shot RL method that uses expectile-based value regularization, a task-conditioned diffusion policy with rejection sampling, and attention-based F/B networks, evaluated on ExORL and D4RL Kitchen.
Problem
Forward-Backward (FB) zero-shot RL methods learn biased, inconsistent successor measures from offline data due to limited expressivity of networks and OOD action extrapolation errors during offline pretraining.
Approach
Replaces the policy-based target in FB's TD loss with an expectile-regressed task-conditioned state-value function (in-sample learning), extracts the policy via a task-conditioned diffusion model trained with advantage-weighted regression, and uses self-attention-based forward and backward representation networks.
Key results
- On ExORL RND full datasets, BREEZE reaches 693±16 Walker, 84±14 Jaco, and 725±23 Quadruped whole-performance IQM, versus FB's 661, 32, and 671.
- On 100k-transition ExORL subsets, BREEZE reaches 525±13 on RND Walker versus 264±33 for FB, and 474±21 on RND Quadruped versus 176±123 for FB.
- Ablation: removing the diffusion policy (Gaussian instead) drops Jaco-APS from 132 to 45 and Quadruped-RND from 725 to 530.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add behavior-regularized representation guidance and policy extraction with diffusion | BREEZE · ExORL full-data domains | IQM | Mixed |
| Remove FB enhancement (use the baseline representation architecture) (ablation) | BREEZE · ExORL RND and APS Walker, Jaco, and Quadruped | IQM: 646$718$ vs 693; 80$724$ vs 84; 685$713$ vs 725; 614$748$ vs 637; 82$713$ vs 132; 655$720$ vs 698 | Hurt |
| Replace diffusion policy with a Gaussian policy (ablation) | BREEZE · ExORL RND and APS Walker, Jaco, and Quadruped | IQM: 707$713$ vs 693; 62$77$ vs 84; 530$733$ vs 725; 587$753$ vs 637; 45$750$ vs 132; 568$719$ vs 698 | Mixed |
| Increase the expectile parameter (ablation) | BREEZE behavior-constrained algorithm · Quadruped RND | performance | Helped |
| Set temperature to 0.05 (ablation) | BREEZE behavior-constrained algorithm · Quadruped RND | performance: 0.05 | Helped |
| Scale up the original MLP-based FB networks (ablation) | Vanilla FB · ExORL | performance | No clear effect |
Surprising findings
- Simply scaling up the original MLP-based FB networks did not yield measurable gains, whereas the attention-based architecture did.
- Naively adding CQL-style value constraints (MCFB/VCFB) gives only marginal improvements over vanilla FB on several settings.
Limitations
- Increased computational cost from diffusion-based sampling and rejection sampling over K candidate actions.
- Experiments are restricted to state-based tasks in ExORL and Kitchen domains.
Experimental setup
Datasets: ExORL D4RL kitchen-mixed D4RL kitchen-partial
Benchmarks: ExORL D4RL Franka Kitchen
Metrics: IQM return normalized score
Baselines: SF-LAP FB VCFB MCFB HILP
Builds on: Forward-Backward representations IQL Diffusion policy weighted regression (IDQL) MCFB/VCFB HILP
Closest papers by meaning
- Zero-Shot Reinforcement Learning from Low Quality Data (NeurIPS 2024 · similarity 0.89)
- Zero-Shot Reinforcement Learning via Function Encoders (ICML 2024 · similarity 0.89)
- Distributional Successor Features Enable Zero-Shot Policy Optimization (NeurIPS 2024 · similarity 0.88)
- VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Provable Zero-Shot Generalization in Offline Reinforcement Learning (ICML 2025 · similarity 0.87)
- Zero Shot Generalization of Vision-Based RL Without Data Augmentation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.