Research map / Reinforcement learning / Visual and representation RL

GoalLadder: Incremental Goal Discovery with Vision-Language Models

NeurIPS 2025 · Reinforcement learning › Visual and representation RL · approach: Vision-based robot RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GoalLadder, which trains RL agents from a single language instruction by using a VLM (Gemini 2.0 Flash) to incrementally discover and ELO-rank candidate goal images, then rewards agents by distance to the top-rated goal in a VAE embedding space.

Problem

Specifying RL rewards from a natural language instruction in visual environments is hard, since existing LLM/VLM-based reward methods are noisy, need many feedback queries, or require non-visual state or environment modifications.

Approach

Incrementally discover candidate goal states via pairwise VLM comparisons, maintain their utility with an ELO rating system that absorbs noisy VLM judgments, and define rewards as Euclidean distance to the top-rated goal in a VAE latent space trained on unlabelled agent observations.

Key results

• GoalLadder achieves a mean success rate of about 95% across tasks versus about 45% for the next best competitor (RL-VLM-F).
• On Metaworld, GoalLadder solves tasks after about 4500 VLM queries, while PEBBLE with ground-truth preferences needs about 15000 labels.
• Ablation on Drawer Open: with ELO final success 0.97 vs 0.20 without ELO (greedy rating).

Measured effects reported in the paper

Surprising findings

• GoalLadder convincingly surpasses the ground-truth-reward oracle on Drawer Open, as the oracle sometimes learns to reach the drawer without pulling the handle.
• A pretrained DINOv2 or CLIP encoder performed worse than a small VAE trained on environment observations for the embedding-based reward.

Limitations

• Assumes task progress or success can be identified from a single image, limiting it to static goals.
• Relies on visual feature similarity for reward definition, which may be a weak proxy for task progress.
• Evaluated on only seven environments due to VLM query costs.

Experimental setup

Datasets: agent-collected unlabelled visual observations (for VAE)

Benchmarks: CartPole MountainCar Metaworld Drawer Open Metaworld Drawer Close Metaworld Sweep Into Metaworld Window Open Metaworld Button Press

Metrics: success rate final success rate sample efficiency (VLM queries)

Baselines: Ground-truth reward (Oracle) VLM-RM RoboCLIP RL-VLM-F PEBBLE

Builds on: SAC PEBBLE RL-VLM-F VLM-RM ELO rating system

Closest papers by meaning

• Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.91)
• SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
• Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.90)
• Bridging Environments and Language with Rendering Functions and Vision-Language Models (ICML 2024 · similarity 0.89)
• NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
• RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.89)