Research map / Vision-language and multimodal / Multimodal LLMs

iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes iSHIFT, a 2.5B GUI agent that uses latent thinking tokens and special perception tokens to implicitly decide between a fast path and a slow DINO-based high-resolution perception path for each action.

Problem

GUI agents either spend uniform compute on all actions or rely on external controllers or explicit text generation to allocate perception and reasoning, which adds latency and complexity while balancing efficiency and grounding precision.

Approach

Integrate continuous latent thinking tokens (<bot>...<eot>) and latent perception tokens (<bop>, <ctrl>, <eop>) into a Qwen2-VL-2B so the model itself decides, via generated tokens, whether to activate a lightweight DINO cross-attention perception module; training data is labeled fast/slow by a rule-based classifier on action type.

Key results

• Achieves 76.34% average AMS on AITW with a 2.5B model, versus 74.27% for AutoGUI (4.5B) and 76.88% for CogAgent (18B).
• Reaches 73.97% success rate on GUI Odyssey, above Aguvis (63.8%, 7B), and 87.7% on Android Control Low.
• Ablation: removing the adaptive switch (slow-only with LTT) drops AMS from 76.34 to 74.58; baseline Qwen2-VL-2B is 67.24.

Surprising findings

• Forcing the slow path for every action performs worse than adaptive switching (74.58 vs 76.34), and the slow path is also slower in runtime.
• Implicit latent thinking outperforms explicit chain-of-thought by 8-11 points average when trained on AITZ and tested on AITW, using far fewer tokens.

Limitations

• Fast/slow labels are assigned by a rule-based if-else classifier on action type rather than learned from perception need.
• Occasional mismatches: selects fast path for tasks needing detail or slow path for simple ones; localization can be slightly off.
• Code and data are promised only upon acceptance; no release at time of writing.

Experimental setup

Datasets: AITW AITZ (Android in the Zoo) Android Control GUI Odyssey GUIAct AutoUI splits

Benchmarks: AITW Android Control GUI Odyssey GUIAct

Metrics: action matching score success rate type accuracy

Baselines: CogAgent SeeClick ShowUI AutoGUI OS-Atlas Aguvis GUICourse TongUI CoCo-Agent MobileAgent GPT-4o OmniParser

Builds on: Qwen2-VL DINOv2 Coconut (latent chain-of-thought) ReAct AutoUI

Closest papers by meaning

• Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.89)
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.88)
• GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents (AAAI 2026 · similarity 0.88)
• GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding (ICML 2026 · similarity 0.87)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.87)
• VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.86)