Research map / Reinforcement learning / Visual and representation RL
Code as Reward: Empowering Reinforcement Learning with VLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLM-CaR, which prompts a VLM (GPT-4 web interface) to decompose a task into sub-tasks and generate code reward functions, verifies them with expert and random trajectories, and uses them as dense auxiliary rewards for RL.
Problem
Defining dense, reliable reward functions for RL in image-based environments is costly, and directly querying VLMs for rewards at every step is computationally expensive, unreliable, and not interpretable.
Approach
Use a VLM once, zero-shot and via sequential prompting, to generate executable Python programs for sub-task completion checks and reward functions from initial and goal images; verify them automatically with expert and random trajectories; then use them as auxiliary dense rewards during RL training, framed as option termination conditions.
Key results
- On MiniGrid, VLM-CaR auxiliary rewards let PPO solve DoorKey-8x8, which standard sparse-reward PPO fails to do, with 3 seeds and 1M steps.
- On PandasGym, dense VLM-CaR rewards generally yield higher final-task success rates than sparse environment rewards with TQC across 5 seeds.
- On CLIPort, the inferred reward separates random, novice, sub-optimal and expert policies, with sub-optimal and expert close on put-blocks-in-bowls and packing-shapes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VLM-CaR-generated auxiliary rewards instead of only sparse environment rewards | PPO · MiniGrid tasks | episodic mean reward | Helped |
| Use VLM-CaR-generated auxiliary rewards instead of only sparse environment rewards | PPO · MiniGrid DoorKey8x8 | task completion | Helped |
| Use VLM-CaR-generated dense rewards instead of sparse environment rewards | TQC · PandasGym PandasPush and PandasSlide | success rate | Helped |
| Use VLM-CaR-generated dense rewards instead of sparse environment rewards | TQC · PandasGym PandasPush and PandasSlide | sample efficiency | Helped |
| Evaluate policies using VLM-CaR-generated reward functions | CLIPort policies · CLIPort separating-piles, put-blocks-in-bowls, and packing-shapes | reward separation by policy skill | Mixed |
Surprising findings
- Generated programs often failed verification at first, yet VLMs produced working scripts within a few attempts (e.g., 11 failed attempts for one MiniGrid-like object in packing-shapes).
- The VLM spontaneously proposed a pathfinding algorithm such as A* for reaching goals in MiniGrid.
Limitations
- Requires manual assembly of VLM-generated sub-task scripts into a single reward class, so the pipeline is not fully automated.
- Object identification relies on colour and shape heuristics and struggles with similar colours, shadows, occlusion, and low resolution (e.g., MetaWorld).
- Results depend on a web interface with timeouts; fixed colours and shapes were used in CLIPort tasks.
Experimental setup
Datasets: Gym-MiniGrid PandasGym CLIPort simulated environments (separating-piles, put-blocks-in-bowls, packing-shapes)
Benchmarks: Gym-MiniGrid DoorKey and Unlock PandasGym PandasPush and PandasSlide CLIPort separating-piles, put-blocks-in-bowls, packing-shapes
Metrics: episodic mean reward success rate sample efficiency number of failed VLM attempts regenerations
Baselines: sparse environment reward PPO with sparse reward TQC with sparse reward
Builds on: options framework proximal policy optimization CLIPort Code as Policies
Closest papers by meaning
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.93)
- Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models (ICML 2025 · similarity 0.91)
- VLMs-Guided Representation Distillation for Efficient Vision-Based Reinforcement Learning (CVPR 2025 · similarity 0.91)
- SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning (ICML 2026 · similarity 0.91)
- COVR: Collaborative Optimization of VLMs and RL Agent for Visual-Based Control (AAAI 2026 · similarity 0.91)
- FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.