Research map / Vision-language and multimodal / CLIP and zero-shot
Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses pretrained CLIP as a zero-shot reward model (VLM-RM) with optional goal-baseline regularization to train RL agents (SAC/DQN) on classic control and MuJoCo humanoid tasks specified by one text prompt.
Problem
Reward specification in vision-based RL is either manual and infeasible or requires expensive human feedback to learn reward models; the paper asks whether pretrained VLMs can serve as zero-shot, language-specified reward functions.
Approach
Use the cosine similarity between a CLIP image embedding of the current state and a CLIP text embedding of a task prompt as the RL reward, with no fine-tuning, and optionally project out irrelevant embedding directions using a neutral baseline prompt (goal-baseline regularization).
Key results
- Five of eight humanoid tasks (kneeling, lotus position, standing up, arms raised, doing splits) reached 100% human-evaluated success rate with the ViT-bigG-14 CLIP reward model.
- Kneeling succeeded only with the largest OpenCLIP ViT-bigG-14 model; smaller models (up to ViT-H-14) achieved 0% success, showing a sharp phase transition.
- EPIC distance to human labels decreases roughly log-linearly with CLIP model size, and textured environments are critical for the CLIP reward to work.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add realistic textures to MountainCar observations (ablation) | CLIP reward model · MountainCar | task success rate | Helped |
| add goal-baseline regularization to the CLIP reward (ablation) | VLM-RM · CartPole | reward landscape quality | No clear effect |
| add goal-baseline regularization to the CLIP reward (ablation) | VLM-RM · humanoid kneeling | EPIC distance to human labels | Helped |
| add realistic textures to the humanoid environment (ablation) | VLM-RM · Humanoid-v4: kneeling | reward model quality | Helped |
| reposition the camera to give a better view of the humanoid (ablation) | VLM-RM · Humanoid-v4: kneeling | reward model quality | Helped |
| increase CLIP model scale | VLM-RM · Humanoid-v4: kneeling | success rate: 0% success rate for all smaller models -> successfully learn the kneeling task with VIT-bigG-14 | Helped |
| add goal-baseline regularization (ablation) | VLM-RM · humanoid tasks that failed with α=0 | task performance | No clear effect |
Surprising findings
- Default MountainCar without custom textures could not be solved with the CLIP reward, while textured rendering enabled 100% success.
- Goal-baseline regularization did not improve performance on the failing humanoid tasks (hands on hips, arms crossed, standing on one leg).
Limitations
- Failed on 3 of 8 humanoid tasks (standing on one leg, hands on hips, arms crossed), attributed to CLIP capability limits and physics constraints.
- Human evaluation done by a single author, which may be biased; success depends on visually realistic environments.
- Goal-based single-image rewards cannot specify temporal or multi-step tasks.
Experimental setup
Datasets: MuJoCo Humanoid-v4 environment CartPole and MountainCar (OpenAI Gym) LAION-5B (CLIP pretraining, not used directly)
Benchmarks: CartPole MountainCar Humanoid-v4 (kneeling, lotus position, standing up, arms raised, doing splits, hands on hips, arms crossed, standing on
Metrics: human-evaluated success rate ground truth episode reward EPIC distance
Baselines: CLIP RN50 OpenCLIP ViT-L-14 OpenCLIP ViT-H-14 ground truth environment reward
Builds on: CLIP Soft Actor-Critic DQN stable-baselines3 EPIC distance
Closest papers by meaning
- Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.93)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Bridging Environments and Language with Rendering Functions and Vision-Language Models (ICML 2024 · similarity 0.91)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.90)
- Unified Reinforcement and Imitation Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.