Research map / Vision-language and multimodal / Robot action and navigation
OS-ATLAS: Foundation Action Model for Generalist GUI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a cross-platform GUI grounding data synthesis toolkit and 13M-element corpus, pretrains open VLMs (Qwen2-VL-7B, InternVL-2-4B) on it, and fine-tunes a unified-action-space GUI action model (OS-Atlas) that beats GPT-4o and prior GUI grounding models on grounding and OOD agent benchmarks.
Problem
Open-source VLMs perform poorly at GUI grounding and out-of-distribution GUI agent tasks, partly due to lack of large multi-platform GUI screenshot/grounding data and conflicting action naming across datasets.
Approach
A multi-platform GUI grounding data synthesis toolkit (web rendering, A11y-tree sampling on Windows/macOS/Linux/Android) yielding 13M+ elements used for grounding pretraining, followed by action fine-tuning under a unified action space (basic click/type/scroll plus custom actions) that resolves cross-dataset action naming conflicts.
Key results
- OS-Atlas-Base-7B reaches 82.47% average grounding accuracy on ScreenSpot without a planner, versus 73.30% for UGround-7B and 53.40% for SeeClick.
- On zero-shot OOD agent benchmarks, OS-Atlas-7B reaches 59.15 SR on OmniAct-Web and 56.73 SR on OmniAct-Desktop, above GPT-4o's 34.06 and 50.67.
- With GPT-4o as planner on OSWorld, OS-Atlas-Base-7B grounding raises average success to 14.63% versus 5.03% for GPT-4o alone and 9.21% with SeeClick.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GUI grounding pre-training to the original VLM (ablation) | OS-Atlas action model · Zero-shot OOD desktop and web tasks | performance | Helped |
| Use a unified action space during fine-tuning (ablation) | OS-Atlas action model · Zero-shot OOD tasks across three platforms | performance | Helped |
| Remove instruction grounding data from grounding pre-training (ablation) | OS-Atlas-Base · ScreenSpot | grounding accuracy | No clear effect |
| Train grounding pre-training on web data only, excluding mobile and desktop data (ablation) | OS-Atlas-Base · ScreenSpot, mobile and desktop platforms | grounding accuracy | Hurt |
| Scale grounding pre-training data | OS-Atlas-Base-4B · ScreenSpot | grounding accuracy and IoU | Helped |
| Use OS-Atlas-Base-7B instead of SeeClick as GPT-4o's grounding module | GPT-4o · OSWorld | successful rate (Avg.): 9.21 -> 14.63 | Helped |
| Use OS-Atlas-7B rather than GPT-4o | GUI action model · GUI-Act-Web, zero-shot OOD | SR: 41.84 -> 57.02 | Helped |
| Use OS-Atlas-7B rather than GPT-4o | GUI action model · OmniAct-Web, zero-shot OOD | SR: 34.06 -> 59.15 | Helped |
| Use OS-Atlas-7B rather than GPT-4o | GUI action model · OmniAct-Desktop, zero-shot OOD | SR: 50.67 -> 56.73 | Helped |
| Use OS-Atlas-7B rather than GPT-4o | GUI action model · AndroidControl-Low, zero-shot OOD | SR: 28.39 -> 50.94 | Helped |
| Use OS-Atlas-7B rather than GPT-4o | GUI action model · AndroidControl-High, zero-shot OOD | SR: 21.17 -> 29.83 | Helped |
| Use OS-Atlas-7B rather than GPT-4o | GUI action model · GUI-Odyssey, zero-shot OOD | SR: 5.36 -> 26.96 | Helped |
Surprising findings
- Pre-training solely on web grounding data fails to generalize to mobile and desktop platforms.
- The InternVL-2-4B base model scores very low on ScreenSpot grounding (4.32 average) despite being a general VLM, showing general VLMs lack GUI grounding.
Limitations
- Desktop agent training data is limited; agent fine-tuning used only three datasets to keep others for OOD testing.
- ScreenSpot accuracy metric does not capture fine-grained grounding errors, motivating IoU and better benchmarks.
Experimental setup
Datasets: FineWeb-filtered web screenshots Windows/Linux/MacOS/Pixel6 synthetic desktop and mobile data SeeClick AMEX UIbert Mind2Web AITZ AndroidControl Wave-UI ScreenSpot GUI-Act-Web OmniAct
Benchmarks: ScreenSpot ScreenSpot-V2 OSWorld GUI-Act-Web OmniAct-Web OmniAct-Desktop AndroidControl-Low AndroidControl-High GUI-Odyssey
Metrics: grounding accuracy IoU success rate (SR) action type exact match grounding accuracy in agent tasks task success rate
Baselines: GPT-4o SeeClick UGround-7B CogAgent Fuyu Qwen2-VL-7B InternVL-2-4B
Builds on: SeeClick Set-of-Mark prompting AnyRes
Closest papers by meaning
- GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents (NeurIPS 2025 · similarity 0.91)
- Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (ICLR 2025 · similarity 0.90)
- Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis (NeurIPS 2025 · similarity 0.90)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.89)
- WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments (ICML 2026 · similarity 0.89)
- TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.