Research map / Vision-language and multimodal / Multimodal LLMs

VLMs-Guided Representation Distillation for Efficient Vision-Based Reinforcement Learning

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Distills semantic saliency signals from VLM prompting (Qwen-VL-Chat reasoning plus CLIP/CRIS referring) into a compact visual encoder for SAC-based vision RL on CARLA, with SSL auxiliary losses.

Problem

Vision-based RL is sample-inefficient and lacks task-relevant semantic constraints when learning state representations from self-exploration data alone; existing VLM uses either rely on VLM decisions or fine-tune with unstable TD gradients.

Approach

Use VLMs only for representation supervision: a reasoning VLM with chain-of-thought prompts outputs a semantic category, a referring model (CLIP/CRIS) turns it into a convolution kernel and saliency map that supervise the compact encoder, refined by transition and reward SSL heads; the policy is trained separately with SAC-style TD loss and VLMs are discarded at test time.

Key results

• On CARLA Highway, DGC reaches episode return 287±47 and driving distance 263±89 vs. 256±51 for MLR and 241±67 for full CRIS fine-tuning (best baseline).
• Removing the saliency feature-map loss Lf drops performance below 240 in the #HW scenario, per the component ablation.
• Frozen CLIP-based representation reaches only 160±64 episode return on #HW, versus 287±47 for DGC.

Measured effects reported in the paper

Surprising findings

• The VLM-based Executor that uses VLM outputs directly for control performs poorly, supporting the claim that VLMs are weaker at fine-grained decision-making than at representation.
• Feeding frozen CLIP image features directly with an FC layer stabilizes around 150 return, below several SSL baselines.

Limitations

• Relies on expensive VLM queries during training, with the reasoning and referring models run on sampled replay batches.
• Evaluated only on CARLA driving scenarios with a single simulator benchmark.
• Reported results are on a single benchmark with modest variance overlap among the top methods.

Experimental setup

Datasets: CARLA simulator interaction data

Benchmarks: CARLA Jaywalk CARLA Highbeam CARLA Highway

Metrics: episode return driving distance

Baselines: SAC DrQ DeepMDP CURL DBC MLR VLM-based Executor VLM-based PFT VLM-based RFT

Builds on: SAC DrQ CURL DBC DeepMDP CRIS

Closest papers by meaning

• COVR: Collaborative Optimization of VLMs and RL Agent for Visual-Based Control (AAAI 2026 · similarity 0.91)
• Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.91)
• Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
• SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning (ICML 2026 · similarity 0.89)
• Empowering Small VLMs to Think with Dynamic Memorization and Exploration (ICLR 2026 · similarity 0.89)
• Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs (CVPR 2026 · similarity 0.89)