Research map / Multi-agent systems and games / LLM agents

CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

ICML 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a procedurally generated dynamic CAPTCHA system and reasoning-action plus self-correction training data to fine-tune Qwen3-VL into ReCAP, a native GUI agent that solves CAPTCHAs and largely retains general GUI performance.

Problem

Native end-to-end GUI agents perform poorly on interactive CAPTCHA solving, while existing specialized CAPTCHA solvers are standalone and do not transfer to general GUI tasks.

Approach

Use ground-truth metadata from a dynamic CAPTCHA renderer to have an expert VLM write answer-conditioned reasoning traces for solutions and retrospective reasoning for failed student attempts, then fine-tune Qwen3-VL with a loss that reweights reasoning and action tokens.

Key results

• ReCAP-32B (Instruct) reaches 84.20% overall solve rate on the dynamic CAPTCHA test set vs 39.30% for Qwen3-VL-32B-Instruct, and above GPT-5.4 (56.80%) and Gemini-3-Flash-Preview (59.90%).
• On the 26-type real-world CAPTCHA benchmark, ReCAP-8B (Instruct) and ReCAP-32B (Instruct) improve over their Qwen3-VL Instruct baselines by 14.50 and 11.42 percentage points on average, zero-shot.
• Ablations on Qwen3-VL-8B-Thinking: reasoning data raises overall SR from 66.40% to 71.90%; self-correction traces raise it from 62.90% to 71.90%.

Surprising findings

• Halligan remains stronger on several real-world rotation and pattern CAPTCHAs (e.g., baidu/rotate 79 vs 21-22, geetest/iconcrush 70 vs 8).
• ReCAP-8B degrades on ScreenSpot-V2 and UI-Vision Element after CAPTCHA fine-tuning, while 32B variants largely preserve general GUI performance.

Limitations

• Relies on synthetic CAPTCHAs; real-world transfer is mixed across types.
• Reasoning traces are answer-conditioned and not verified as faithful explanations.
• Does not model human behavioral biometrics such as mouse speed or hesitation.

Experimental setup

Datasets: Aguvis AgentNet

Benchmarks: dynamic CAPTCHA system (1,000 held-out instances, 7 types) Halligan real-world CAPTCHA benchmark (26 types) Android Control ScreenSpot-V2 Multimodal-Mind2Web UI-Vision

Metrics: solve rate (SR %) average steps in successful cases accuracy IoU

Baselines: UI-TARS-1.5-7B Qwen3-VL-8B-Thinking Qwen3-VL-32B-Thinking Qwen3-VL-8B-Instruct Qwen3-VL-32B-Instruct OpenAI CUA GPT-5.4 Gemini-3-Flash-Preview Halligan

Builds on: STaR Qwen3-VL Aguvis AgentNet Halligan

Closest papers by meaning

• Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents (NeurIPS 2025 · similarity 0.91)
• Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense (ICML 2026 · similarity 0.89)
• Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiation (ICLR 2026 · similarity 0.88)
• MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks (AAAI 2026 · similarity 0.86)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent (CVPR 2025 · similarity 0.85)
• Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.85)