Research map / Vision-language and multimodal / Multimodal LLMs
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper presents Ferret-UI 2, a multimodal LLM for universal UI understanding across iPhone, Android, iPad, webpage, and AppleTV, as described in an author rebuttal rather than the method itself.
Problem
Building a generalist model for UI understanding across platforms is hard due to platform diversity, resolution variation, and limited data.
Approach
Adaptive N-gridding for high-resolution UI perception and curated data processing that reduces cross-platform domain gaps so one model handles multiple platforms.
Key results
- Ferret-UI 2 scores 89.73 GPT-4o score versus 45.81 for Ferret-UI on a UI understanding evaluation (Table 2, per rebuttal).
- A UI navigation agent built on Ferret-UI reaches 53.2% task completion versus 43.7% for a GPT-4o SoM agent (preliminary, per rebuttal).
- Cross-platform zero-shot transfer is strong among iPhone, iPad, and Android, but limited from AppleTV to mobile platforms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ferret-UI One instead of Ferret-UI | Ferret-UI · Table 2 evaluation | GPT-4o score: 45.81 -> 89.73 | Helped |
| Use Ferret-UI One as the UI navigation agent instead of GPT4o-SoM | UI navigation agent · UI navigation | task completion rate: 43.7 -> 53.2 | Helped |
| Train a single model using curated, uniformized data across platforms | Ferret-UI One · Cross-platform UI understanding | zero-shot transfer | Helped |
| Transfer UI knowledge from iPhone, iPad, or Android to another of these platforms | Ferret-UI One · Cross-platform transfer among iPhone, iPad, and Android | transfer performance: >65 | Helped |
| Transfer UI knowledge from AppleTV to mobile platforms | Ferret-UI One · AppleTV-to-mobile transfer | transfer performance | Hurt |
Surprising findings
- Limited zero-shot transfer from AppleTV to mobile platforms.
Limitations
- Model release status is undisclosed in the rebuttal; code and weights for Ferret-UI 2 not confirmed.
- Agent planning and self-reflection are weak; most navigation failures stem from planning errors.
- Model focuses on single-step understanding and interaction rather than full navigation.
Experimental setup
Benchmarks: GUI-World
Metrics: GPT-4o score task completion rate
Baselines: Ferret-UI GPT-4o SoM agent CogAgent
Builds on: Ferret-UI CogAgent
Closest papers by meaning
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding (ICLR 2025 · similarity 0.86)
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.86)
- GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior (NeurIPS 2025 · similarity 0.85)
- CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design (AAAI 2026 · similarity 0.85)
- Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining (ICML 2026 · similarity 0.85)
- UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.