Research map / Large language models / Agents, code and math
Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Open CaptchaWorld, a web-based benchmark of 225 interactive CAPTCHA puzzles across 20 types plus a CAPTCHA Reasoning Depth metric, and shows MLLM browser-use agents solve at most 40% versus 93.3% for humans.
Problem
MLLM web agents are blocked by CAPTCHAs, yet existing agent benchmarks filter them out and no benchmark tests closed-loop interactive CAPTCHA solving by multimodal agents.
Approach
A web-based interactive CAPTCHA benchmark with 20 types and a task-agnostic CAPTCHA Reasoning Depth metric counting human-observed atomic cognitive and motor steps, evaluated with Browser-Use agents driven by different MLLM backbones in zero-shot settings.
Key results
- Openai-o3 achieves the highest agent pass@1 at 40.0%, versus 93.3% for humans.
- GPT-4.1 and Gemini2.5-Pro reach 25.0% pass@1; Claude-3.7-Sonnet and DeepSeek-V3 reach 20.0%.
- Average human-estimated CAPTCHA Reasoning Depth is 2.94 with standard deviation 0.92.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Browser Use + Openai-o3 instead of human solving | Open CaptchaWorld · Open CaptchaWorld | Pass@1 (%): 93.30 -> 40.0 | Hurt |
| Use Openai-o3 instead of GPT-4o as the Browser Use agent backbone | Browser Use agent · Open CaptchaWorld | Pass@1 (%): 5.7 -> 40.0 | Helped |
| Use Openai-o3 instead of GPT-4o as the Browser Use agent backbone | Browser Use agent · Open CaptchaWorld | Cost ($): 25.8 -> 66.4 | Hurt |
| Use Openai-o1 instead of Openai-o3 as the Browser Use agent backbone | Browser Use agent · Open CaptchaWorld | Pass@1 (%): 40.0 -> 5.0 | Hurt |
Surprising findings
- Openai-o1 has the lowest success rate (5.0%) while being the most expensive run ($94.6).
- Models tend to overthink, over-segmenting tasks into fine-grained steps compared with humans' compressed reasoning depth estimates.
Limitations
- Benchmark is small (225 CAPTCHAs), with a modest number of instances per type.
- Evaluation is zero-shot with a single browser-use agent framework and a single run per model.
Experimental setup
Benchmarks: Open CaptchaWorld
Metrics: Pass@1 success rate cost (USD) CAPTCHA Reasoning Depth
Baselines: Human GPT-4o GPT-4.1 Claude-3.7-Sonnet Claude-3.5-Sonnet Claude-3.5-Haiku Gemini2.5-Pro Openai-o1 DeepSeek-V3
Builds on: Browser Use
Closest papers by meaning
- CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training (ICML 2026 · similarity 0.91)
- Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense (ICML 2026 · similarity 0.90)
- Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiation (ICLR 2026 · similarity 0.90)
- MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks (AAAI 2026 · similarity 0.89)
- Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.87)
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.