Research map / Reinforcement learning / Visual and representation RL

Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning

ICML 2025 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoSo, an online RL fine-tuning method for VLM agents that uses counterfactual token-level causal weights to reweight entropy regularization, focusing exploration on action-critical tokens; it improves DigiRL and RL4VLM on AitW, Gym Cards, and ALFWorld.

Problem

Online RL fine-tuning of VLM agents is inefficient because the open-ended textual output space is huge and only a small subset of tokens affects the post-processed executable action, so uniform entropy exploration wastes effort.

Approach

Compute per-token causal weights via counterfactual token nullification measured on a surrogate SCM (a BERT-based classifier), and use these weights to scale per-token entropy in the soft RL objective, so exploration concentrates on action-critical tokens instead of uniform uncertainty over all tokens.

Key results

• On AitW, CoSo reaches 72.9% average success, reported as a 12.3% improvement over DigiRL (64.9%).
• On Gym Cards, CoSo achieves 49.3% average vs. 45.1% for RL4VLM, a 9.3% gain; on ALFWorld it reaches 26.5% vs. 22.7%.
• Counterfactual analysis finds under 10% of tokens are action-critical while over 80% have weights below 0.2.

Measured effects reported in the paper

Surprising findings

• Naive entropy regularization over all tokens provides only marginal improvement over standard RL on WebShopping and NumberLine.
• On the text-only AlfredTWEnv with a Qwen2.5-7B LLM agent, CoSo improves average success from 32.8% to 39.6% over RL4VLM.

Limitations

• Longest utterances in experiments are under 300 tokens; effectiveness on ultra-long chain-of-thought is unexplored.
• Gains are reported over only a few seeds (3) with large standard deviations on AitW, so differences may be noisy.
• Theoretical guarantees rely on standard soft-RL assumptions and a bounded-entropy argument rather than analysis of the learned SCM's accuracy.

Experimental setup

Datasets: Android-in-the-Wild (AitW) General Android-in-the-Wild (AitW) WebShopping Gym Cards (NumberLine, EZPoints, Points24, Blackjack) ALFWorld AlfredTWEnv

Benchmarks: AitW General AitW WebShopping Gym Cards ALFWorld AlfredTWEnv

Metrics: success rate (%)

Baselines: Gemini 1.5 Pro GPT-4V AppAgent CogAgent AutoUI + SFT Online Filtered BC DigiRL RL4VLM CNN + RL LLaVA-1.6 + SFT

Builds on: DigiRL RL4VLM soft actor-critic PPO AWR

Closest papers by meaning

• COVR: Collaborative Optimization of VLMs and RL Agent for Visual-Based Control (AAAI 2026 · similarity 0.87)
• A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.87)
• Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach (ICML 2025 · similarity 0.86)
• Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining (ICLR 2025 · similarity 0.86)
• Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors (NeurIPS 2026 · similarity 0.86)