Research map / Reinforcement learning / RL from feedback
FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FuRL, which fine-tunes lightweight projection heads on frozen VLM (LIV/CLIP) embeddings for reward alignment and uses relay RL with a second SAC agent to escape local minima from fuzzy VLM rewards in sparse-reward Meta-world tasks.
Problem
Zero-shot VLM rewards (cosine similarity of CLIP-style embeddings) are fuzzy and misaligned with task progress, so using them as dense rewards in sparse-reward online RL can trap agents in local minima.
Approach
Freeze the pre-trained VLM and learn two small MLP projection heads for text and image embeddings with a ranking loss driven by sparse task success (reward alignment), combined with relay RL that alternates rollouts from a VLM policy and a SAC policy to find first successes.
Key results
- On sparse MT10 with fixed goals, FuRL averages 76% success vs. 37% for SAC, 54% for Relay, and 28% for LIV, and VLM-only rewards (VLM-RMs) solve no tasks.
- With random goals, FuRL averages 64.6% success vs. 49.8% for SAC and 57.4% for Relay.
- With CLIP as the VLM, CLIP-FuRL averages 70% vs. 38% for the CLIP baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FuRL instead of Relay, adding reward alignment | SAC with LIV rewards · Meta-World MT10, fixed goal | average success rate (%): 54.0 -> 76.0 | Helped |
| Use FuRL instead of Relay, adding reward alignment | SAC with LIV rewards · Meta-World MT10, random goals | average success rate: 57.4 (7.0) -> 64.6 (5.0) | Helped |
| Add FuRL alignment and relay to the CLIP reward baseline | SAC with CLIP rewards · Meta-World MT10, fixed goal | average success rate: 38 (9.8) -> 70 (6.3) | Helped |
| Remove reward alignment (ablation) | FuRL · Meta-World tasks in the ablation study | average success rate and AUC | Hurt |
| Remove Stage 1 of reward alignment (ablation) | FuRL · Meta-World tasks in the ablation study | average success rate and AUC | Hurt |
| Remove Stage 2 of reward alignment (ablation) | FuRL · Meta-World tasks in the ablation study | average success rate and AUC | Hurt |
| Remove relay RL (ablation) | FuRL · button-press-topdown and door-open | success | Hurt |
| Remove the pre-trained VLM representation (ablation) | FuRL · Meta-World tasks in the ablation study | average success rate | Hurt |
| Use a larger VLM reward weight (ablation) | FuRL · drawer-open and button-press-topdown with random goals | learning performance | Hurt |
| Use a smaller VLM reward weight (ablation) | FuRL · drawer-open and button-press-topdown with random goals | learning speed and variance | Mixed |
| Replace the task instruction with a detailed instruction (ablation) | FuRL · button-press-topdown and drawer-open | learning speed | Helped |
| Replace the task instruction with a meaningless instruction (ablation) | FuRL · button-press-topdown and drawer-open | learning performance | Hurt |
Surprising findings
- Naively adding the LIV VLM reward to SAC performs worse than plain SAC (28% vs. 37% average success).
- VLM-only reward policies (VLM-RMs, VLM-RMs-GB) fail to solve any task.
Limitations
- Experiments only in simulation (Meta-world), with no real-world robot validation.
- Relay RL requires maintaining a second policy, increasing computation.
- Reward alignment is trained per task and single-task language instructions are used.
Experimental setup
Datasets: Meta-world MT10
Benchmarks: Meta-world MT10 sparse fixed-goal Meta-world MT10 sparse random-goal
Metrics: success rate AUC
Baselines: SAC VLM-RMs VLM-RMs-GB LIV LIV-Proj Relay DrQ CLIP
Builds on: LIV VLM-RMs Jump-start RL CLIP SAC
Closest papers by meaning
- Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.90)
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models (ICML 2025 · similarity 0.88)
- Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.88)
- SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning (ICML 2026 · similarity 0.87)
- What Can RL Bring to VLA Generalization? An Empirical Study (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.