Research map / Vision-language and multimodal / Robot action and navigation
Bridging Environments and Language with Rendering Functions and Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decomposes language-conditioned control into VLM-based text-to-goal configuration search (multiview, distilled, finetuned) plus a pretrained goal-conditioned policy, evaluated on 256 tasks in a modified Humanoid.
Problem
Learning language-conditioned agents without annotated text-environment data; single-task RL with VLM rewards is costly and MTRL needs carefully designed task corpora and generalizes unreliably.
Approach
Use rendering functions and VLM image-text similarity to find high-scoring environment configurations for a text (via retrieval, distilled-model gradient finetuning, and exact re-scoring), then reach them with a task-agnostic goal-conditioned policy; enhancements include multiview scoring and a distilled embedding model.
Key results
- GCRL-based LCA (finetuned + selected) exceeds MTRL (test) approximate VLM return on 203 of 256 tasks (exact: 205 for GCRL-S).
- Distilled model reduces configuration-embedding computation time by 40,000x (4.8e-4 s vs 21.1 s for 256 three-view embeddings).
- Multiview retrieval gives configurations that score robustly across unseen viewpoints, while single-view retrieval scores drop significantly from other views.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an embedding-diversity dataset rather than random-policy or uniform-sampling data for configuration retrieval (ablation) | Humanoid configuration retrieval · Humanoid (256 tasks) | single-view configuration-text score: 0.6132 -> 0.6260 | Helped |
| Use three retrieval views instead of one (ablation) | Humanoid configuration retrieval · Humanoid, front-view evaluation | single-view configuration-text score: 0.6260 -> 0.6147 | Hurt |
| Use three retrieval views instead of one (ablation) | Humanoid configuration retrieval · Humanoid, mid-left- and left-view evaluation | single-view configuration-text score: 0.6050 -> 0.6138; 0.5835 -> 0.6104 | Helped |
| Add distilled-model finetuning to retrieved configurations (ablation) | Humanoid configuration retrieval · Humanoid (256 tasks) | multiview configuration-text score: 0.6265 -> 0.6289 | Helped |
| Add exact-VLM selection after finetuning (ablation) | Humanoid configuration retrieval · Humanoid (256 tasks) | multiview configuration-text score: 0.6289 -> 0.6343 | Helped |
| Decompose task execution into configuration generation and goal-reaching rather than using MTRL | Zero-shot Humanoid LCA · Humanoid (256 tasks) | exact VLM return: 205 of the 256 tasks | Helped |
| Use distilled-model finetuned goals with GCRL rather than MTRL test policy | Zero-shot Humanoid LCA · Humanoid (256 tasks) | exact VLM return: 198 of the 256 tasks | Helped |
| Use distilled-model finetuned goals with GCRL rather than MTRL test policy | Zero-shot Humanoid LCA · Humanoid (256 tasks) | approximate VLM return: 203 of the 256 tasks | Helped |
| Use finetuned goal configurations instead of STRL trajectories | Humanoid LCA · Humanoid (256 tasks) | approximate configuration-text score: 0.6202; 0.6324 | Helped |
| Distill configuration embeddings into a neural network rather than evaluate with EVA02-E-14+ | VLM-based configuration scoring · Humanoid configuration embeddings | computation speedup: 40 000× | Helped |
Surprising findings
- Configurations retrieved from a single view often look acceptable from the front but are pathological from side views due to occlusion and distance ambiguity.
Limitations
- Restricted to static configurations; dynamic tasks such as juggling or cycling are not covered.
- Current VLMs struggle with compositional relationships between multiple objects.
Experimental setup
Datasets: embedding-diversity configuration dataset (2.5M, sampled) random policy configuration dataset uniform sampling configuration dataset
Benchmarks: 256 textual tasks on modified Humanoid (Gym/MuJoCo)
Metrics: average configuration-text similarity score approximate VLM return best-in-trajectory score number of tasks where LCA beats MTRL (test)
Baselines: STRL MTRL (train) MTRL (test) single-view VLM reward (past work)
Builds on: CLIP Colas et al. 2020 text-to-goal decomposition PPO MuJoCo
Closest papers by meaning
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.91)
- VICO: Visual Environments Co-Evolving for Vision-Language Model Reasoning (NeurIPS 2026 · similarity 0.90)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.89)
- Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.89)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.