Research map / Vision-language and multimodal / Multimodal LLMs

Exposing and Evaluating Hallucinations for GUI Grounding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies confusion and fabricated hallucinations in GUI grounding, builds the bilingual GUI-HalluBench (parsing and hallucination subsets), and eases them with a training-free parsing-guided prompt (PGP) and LoRA fine-tuning on hallucination-labeled data (HFT).

Problem

Existing GUI benchmarks measure general grounding and navigation competence but do not evaluate hallucinations in GUI grounding, such as selecting distractor elements or confidently grounding nonexistent elements.

Approach

Defines two hallucination types (confusion and fabricated) for GUI grounding, links them to parsing deficiencies via a dual-subset benchmark, and proposes parse-then-ground prompting plus LoRA fine-tuning on automatically generated, human-verified hallucination data.

Key results

• Qwen3-VL-8B with HFT improves average score from 66.0% to 73.0% on GUI-HalluBench.
• Training-free PGP raises Qwen3-VL-8B average from 66.0% to 67.7%, with modest gains.
• Gemini-2.0 (Project Mariner) achieves the highest closed-source average at 60.5%, while GPT-4o has only 4.3% element precision on English parsing.

Surprising findings

• Open-source models fine-tuned on Chinese-scale data (Qwen3-VL, InternVL) outperform closed-source models such as GPT-4o and Claude on this benchmark.
• Closed-source models lead neither the parsing nor the hallucination subtask.

Limitations

• Limited coverage of specialized professional domains such as medical, financial, and engineering interfaces.
• Hallucination cases are built on current GUI layouts, so effectiveness may drop for emerging interface styles.
• HFT labels are validated by a single annotator on the 10K training set rather than three.

Experimental setup

Datasets: WidgetCaption UI RefExp RICO-Semantics RICO-SCA Os-Atlas ShowUI GUIEnv SeeClick-Web ScreenQA ShareGPT-Computer LLaVA-Instruct ScreenSpot

Benchmarks: GUI-HalluBench

Metrics: element recall element precision function similarity localization accuracy rejection rate

Baselines: GPT-4o Claude Computer Use Gemini-2.0 (Project Mariner) InternVL3.5-8B Qwen3-VL-8B SeeClick CogAgent OS-Atlas UI-TARS Aguvis UGround JEDI

Builds on: SeeClick ScreenSpot Grounding DINO PaddleOCR

Closest papers by meaning

• Reducing Tool Hallucination via Reliability Alignment (ICML 2025 · similarity 0.87)
• CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification (AAAI 2025 · similarity 0.86)
• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments (ICML 2026 · similarity 0.86)
• HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data (CVPR 2024 · similarity 0.86)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis (NeurIPS 2025 · similarity 0.85)
• MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM (NeurIPS 2025 · similarity 0.85)