Research map / Vision-language and multimodal / Multimodal LLMs
CogAgent: A Visual Language Model for GUI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CogAgent, an 18B visual language model with a low-res plus high-res cross-attention encoder branch for GUI understanding and navigation, which beats LLM-based HTML methods on Mind2Web and AITW using screenshots only.
Problem
Language-only LLM agents struggle with GUIs because they rely on extracted HTML or OCR text and miss visual information, while existing VLMs use low input resolutions that cannot resolve tiny GUI text and icons.
Approach
Add a small high-resolution cross-attention branch (1120x1120 input, EVA2-CLIP-L encoder, 0.30B) to the CogVLM decoder so high-resolution features are fused via cross-attention in each layer, keeping the sequence length of the decoder unchanged and reducing compute versus raising the resolution directly; pair this with GUI-specific pre-training data (CCS400K web screenshots with DOM grounding) and
Key results
- Mind2Web overall step SR of 58.2 vs 54.4 for LLaMA2-70B (HTML input) and 23.9 for CogVLM.
- AITW overall score 76.88 vs 74.27 for Auto-UI, with 93.49 on the Single subset.
- MM-Vet 52.8 zero-shot (vs 36.3 for LLaVA-1.5) and state-of-the-art generalist results on 5 of 6 text-rich VQA benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the high-resolution cross-module at 1120 resolution instead of using the original CogVLM structure at 490 resolution (ablation) | CogAgent · ST-VQA | ANLS: 68.1 -> 78.2 | Helped |
| Add the high-resolution cross-module at 1120 resolution instead of using the original CogVLM structure at 490 resolution (ablation) | CogAgent · DocVQA | ANLS: 57.6 -> 74.1 | Helped |
| Add the high-resolution cross-module at 1120 resolution instead of using the original CogVLM structure at 490 resolution (ablation) | CogAgent · Mind2Web | step SR: 40.7 -> 41.4 | Helped |
| Increase cross-module resolution from 756 to 1120 (ablation) | CogAgent · ST-VQA | ANLS: 73.6 -> 78.2 | Helped |
| Increase cross-module resolution from 756 to 1120 (ablation) | CogAgent · DocVQA | ANLS: 62.3 -> 74.1 | Helped |
| Add OCR pre-training data to image-caption pre-training data (ablation) | CogAgent · ST-VQA | ANLS: 68.1 -> 72.5 | Helped |
| Add OCR pre-training data to image-caption pre-training data (ablation) | CogAgent · Mind2Web | step SR: 38.6 -> 40.7 | Helped |
| Add GUI and grounding pre-training data to caption and OCR data (ablation) | CogAgent · Mind2Web | step SR: 41.4 -> 54.2 | Helped |
| Add GUI and grounding pre-training data to caption and OCR data (ablation) | CogAgent · ST-VQA | ANLS: 78.2 -> 79.4 | Helped |
| Add GUI and grounding pre-training data to caption and OCR data (ablation) | CogAgent · OCR-VQA | EM: 75.9 -> 75.6 | Hurt |
| Use CogAgent instead of LLaVA-1.5 | visual question answering · MM-Vet | GPT-4 score: +16.5 | Helped |
| Use screenshot input with CogAgent instead of HTML input with LLaMA2-70B | Mind2Web · cross-task | step SR: 55.8 -> 62.3 | Helped |
Surprising findings
- In sampled AITW discrepancies, 42% of model answers that disagreed with ground truth were actually alternative correct methods.
- Adding the high-resolution cross-module at 1120 input yields lower FLOPs than CogVLM at 490 input, despite the higher resolution.
Limitations
- Imprecise output coordinates for GUI actions.
- Cannot process multiple images as input.
- AITW evaluation penalizes valid alternative action paths (42% of sampled disagreements were correct).
Experimental setup
Datasets: LAION-2B COYO-700M LAION-115M Nougat arXiv data CCS400K Mind2Web AITW VQAv2 OK-VQA OCR-VQA TextVQA ST-VQA
Benchmarks: VQAv2 OK-VQA OCR-VQA TextVQA ST-VQA ChartQA InfoVQA DocVQA MM-Vet POPE Mind2Web AITW
Metrics: VQA score ANLS EM F1 GPT-4 score step success rate matching score TFLOPs
Baselines: GPT-3.5 GPT-4 Flan-T5-XL LLaMA2-7B LLaMA2-70B Qwen-VL CogVLM Auto-UI PALI-X-55B LLaVA-1.5
Builds on: CogVLM EVA2-CLIP Vicuna Mind2Web AITW
Closest papers by meaning
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.88)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.88)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.87)
- OS-ATLAS: Foundation Action Model for Generalist GUI Agents (ICLR 2025 · similarity 0.86)
- Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.86)
- Generative Visual Code Mobile World Models (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.