Research map / Reinforcement learning / Policy gradient and actor-critic

Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents

ICLR 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Digi-Q trains a VLM-based Q-function offline via representation fine-tuning plus TD-learning on frozen LLaVA features, then extracts a device-control policy with a Best-of-N objective on Android-in-the-Wild.

Problem

Training device-control agents with RL usually needs costly on-policy interaction; the paper asks whether a state-action Q-function learned purely from offline agent trajectories can score candidate actions and yield a better policy for Android device control.

Approach

Fine-tune a VLM on a binary "does this action change the screen" objective to make its features action-aware, freeze it, train Q and V heads with TD-learning on those features, and extract the policy by imitating the best of N sampled actions ranked by Q (only if advantage is positive), avoiding negative gradients.

Key results

• On AitW Web Shopping test, Digi-Q reaches 58.0% success versus 47.6% for DigiRL (offline) and 43.8% for Filtered BC.
• On AitW General test, Digi-Q reaches 71.2% versus 59.0% for offline DigiRL, about 21.2% relative improvement averaged over the two test subsets.
• Ablation: MC-regression critic gets 37.5% and off-the-shelf VLM critic 31.9%, versus 58.0% for full Digi-Q; AWR actor drops to 19.4%.

Measured effects reported in the paper

Surprising findings

• Off-the-shelf VLM representations give only marginal improvement over the behavior policy (31.9% vs 25.0%), and a BLIP-2 + BERT critic collapses toward a state-only value function.
• REINFORCE improves the policy (37.5%) but has the highest variance; AWR is conservative and falls below the behavior policy (19.4%).

Limitations

• Evaluation depends on a proprietary Gemini 1.5 Pro autonomous evaluator and on web environments that changed over time (blocked sites), making numbers non-stationary.
• Using the critic for active online self-improvement is left for future work because it needs a more sophisticated system and robustness as the policy drifts from the data distribution.

Experimental setup

Datasets: Android-in-the-Wild (AitW) Web Shopping Android-in-the-Wild (AitW) General

Benchmarks: Android-in-the-Wild General Android-in-the-Wild Web Shopping

Metrics: task success rate critic accuracy vs human advantage labels token-level KL divergence to behavior policy

Baselines: GPT-4V Set-of-Marks Gemini 1.5 Pro Set-of-Marks AppAgent CogAgent AutoUI Filtered BC DigiRL REINFORCE AWR BLIP-2 + BERT critic Monte-Carlo regression critic

Builds on: ArCHer DigiRL AWR LLaVA-1.5 AutoUI

Closest papers by meaning

• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
• DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agent (ICLR 2025 · similarity 0.87)
• Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.86)
• Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.86)
• O-MAPL: Offline Multi-agent Preference Learning (ICML 2025 · similarity 0.85)
• Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.85)