Research map / Reinforcement learning / Visual and representation RL
RLZero: Direct Policy Inference from Language Without In-Domain Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLZero, which turns a language prompt into a zero-shot policy by imagining a video with a text-conditioned generator, projecting frames to agent observations via SigLIP retrieval, and imitating them in closed form with a pretrained Forward-Backward behavior foundation model.
Problem
Mapping arbitrary natural-language task instructions to executable RL policies without in-domain labeled trajectories, task-specific training, or test-time learning.
Approach
Imagine (text-to-video generation from unsupervised world-model training), project (SigLIP nearest-neighbor retrieval of offline observations), and imitate (closed-form latent inference in a successor-feature BFM using a shaped distribution-matching reward e^nu = rho^E/rho), giving a single policy per environment with no gradient steps at test time.
Key results
- RLZero achieves an 83.2% GPT-4o win rate over the TD3 image-language reward base model across 25 language tasks, versus 76.8% for GenRL and 51.2% for IQL with image-language reward.
- RLZero policies are produced in about 25 seconds with no test-time training, compared with roughly 3-5 hours per task for GenRL on an A40 GPU.
- On 17 cross-embodied video clips, RLZero beats SMODICE with an 80% win rate on Stickman and 100% on HumEnv.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RLZero instead of GenRL | Language-to-behavior across 25 tasks · Language-to-behavior, 25 tasks | GPT-4o win rate: 76.8% -> 83.2% | Helped |
| use RLZero instead of GenRL | Language-to-behavior across 25 tasks · Language-to-behavior, 25 tasks | distribution-matching return: 246.48 -> 295.11 | Helped |
| use RLZero instead of SMODICE | Cross-embodied imitation on Stickman · Stickman | GPT-4o win rate: 80% | Helped |
| use RLZero instead of SMODICE | Cross-embodied imitation on HumEnv · HumEnv | GPT-4o win rate: 100% | Helped |
| remove video imagination and projection and map language embeddings directly to BFM skill embeddings (ablation) | RLZero on Walker · Walker language tasks | GPT-4o win rate against TD3 base model: 91.4% -> 65.71% | Hurt |
| add a discriminator for the imitation reward (ablation) | RLZero on language-to-behavior tasks · Language-to-behavior, 25 tasks | average GPT-4o win rate: 83.2% -> 82.4% | No clear effect |
Surprising findings
- An imagination-free variant that maps text embeddings directly to the latent skill reached a 65.71% win rate on Walker, versus 91.4% for the full pipeline.
- Using a learned discriminator for the distribution ratio did not improve over the discriminator-free shaped reward (82.4% vs 83.2%).
Limitations
- The video generation model is small (about 43M parameters), which limits imagining complex behaviors and makes it sensitive to prompt engineering.
- SigLIP-based retrieval fails with background distractors and with roughly symmetric embodiments.
- Evaluation relies on GPT-4o as a judge and on distribution-matching returns rather than human judgments.
Experimental setup
Datasets: ExoRL RND exploration dataset (Cheetah, Walker, Quadruped, Stickman) YouTube and AI-generated videos for cross-embodiment
Benchmarks: DeepMind Control Suite (Walker, Cheetah, Quadruped, Stickman) HumEnv 3D humanoid
Metrics: GPT-4o pairwise win rate distribution matching return under discriminator reward statistical significance (Mann-Whitney U)
Baselines: IQL with image-language reward TD3 with image-language reward TD3 with video-language reward IQL with video-language reward GenRL SMODICE
Builds on: GenRL Forward-Backward representations ExoRL SMODICE VLM-RL
Closest papers by meaning
- Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data (NeurIPS 2025 · similarity 0.90)
- Text-Aware Diffusion for Policy Learning (NeurIPS 2024 · similarity 0.90)
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.89)
- SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data (ICML 2025 · similarity 0.89)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.