Research map / Vision-language and multimodal / Multimodal LLMs

UI-Lens: Assessing General MLLMs' Potential to Automate UI Display Quality Assurance

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces UI-Lens, an expert-annotated benchmark of Chinese (and planned English) mobile UI screens for detecting six display-defect types, and evaluates nine closed and open MLLMs zero-shot, finding near-random performance on boundary-related defects and weak cross-page inconsistency detection.

Problem

Whether general multimodal LLMs and VLMs can detect UI display defects (text overflow, cropped content, container overlap, missing content, text ellipsis, cross-page text inconsistency) in realistic commercial app interfaces remains systematically unvalidated.

Approach

A benchmark built from real screens of 11 commercial apps across 21 devices, with expert bounding-box annotations, expert-synthesized positives (Figma edits), and a 1:1 positive/negative balance, plus a separate open-ended protocol for sequence-level text inconsistency evaluated with a judge model.

Key results

• Fine-grained boundary tasks are near random: task-average accuracy 50.04% for Text Overflow, 54.07% for Cropped Content, 54.14% for Container Overlap.
• Task-average F1 is 22.19% for Text Overflow, 33.75% for Container Overlap, and 11.44% for Text Inconsistency.
• Best models (Seed1.6-Vison, Gemini-2.5-Pro) reach only about 64%-66% model-average F1; one-shot CoT adds only +0.60 to +2.48 F1 over zero-shot.

Surprising findings

• Models with the highest recall (Gemini-2.5-Pro, 81.41%) have the lowest precision (56.85%), while Seed1.5-VL reaches 75.7% precision but only 27.07% recall.
• Synthetic expert-made positives match real-data accuracy (62.82% vs 61.81%), suggesting synthesized defects are representative.

Limitations

• Expert-annotated, small-scale benchmark with about 1,030 synthetic positives in the Chinese set; the English version is not yet released.
• Sequence-level evaluation relies on a judge model whose identity is not specified in the text.
• Only zero-shot CoT and a few prompting variants tested; no fine-tuning or training-based methods explored.

Experimental setup

Datasets: UI-Lens (Chinese subset)

Benchmarks: UI-Lens

Metrics: accuracy precision recall F1 score

Baselines: Seed1.5-VL Seed1.6-Vison Seed1.6 GPT-5 GPT-4.1 GPT-4o Gemini-2.5-Pro GLM-4.5V Qwen3-VL-235B-A22B

Builds on: chain-of-thought prompting self-correction fine-grained visual prompting

Closest papers by meaning

• CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design (AAAI 2026 · similarity 0.89)
• FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection (CVPR 2026 · similarity 0.86)
• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments (ICML 2026 · similarity 0.84)
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction (ICML 2025 · similarity 0.84)
• Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms (ICLR 2025 · similarity 0.83)
• Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs (CVPR 2026 · similarity 0.83)