Research map / Multi-agent systems and games / LLM agents
Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Next-Gen CAPTCHAs, 27 procedurally generated CAPTCHA families designed to resist GUI agents, with a 519-puzzle benchmark where humans score 98.8% and frontier MLLM agents in Browser-Use score at most 5.9% Pass@1.
Problem
Traditional CAPTCHAs and logic puzzles are being solved by reasoning MLLMs and GUI computer-use agents at high rates, so web defenses need new challenges that humans solve easily but agents fail.
Approach
Design CAPTCHA families that target a human-agent "Cognitive Gap" in interactive perception, memory, decision-making and action (e.g., drag-and-drop, motion-contrast, timed clicks), generated procedurally with rule-based verification, and filter families by agent failure and human success.
Key results
- Humans achieve 98.8% Pass@1 (CS-background, 268 trials, 31.1 s average) while the best MLLM backbone, GPT-5.2-xHigh in Browser-Use, reaches only 5.9% Pass@1.
- Gemini-3-Flash-High Pass@1 rises from 0.6% (minimal thinking) to 3.2% (high), showing thinking more helps only marginally.
- Family-aware hint adaptation raises overall Gemini-3-Flash-High Pass@1 from 3.2% to 6.4%, mainly driven by the Red_Dot family (15% to 100%).
Surprising findings
- Claude Cowork framework reaches 4.44% vs Browser-Use 1.48% and CrewAI 0.00% with the same backbone, showing framework choice matters.
- Higher reasoning-budget and more steps show near-zero or negative correlation with success for Next-Gen families, unlike current CAPTCHAs.
Limitations
- Only 2 of 27 families are non-generated (curated), and filtering used a single lightweight model (Gemini 3 Flash) on a 20-instance pilot set.
- Evaluation uses a subset of models for cost reasons (GPT-5.2 and Claude on 135 puzzles), and results rely on a single run with small per-family sample sizes (often 20 or 5).
- Family-aware hints show a substantial gain on Red_Dot (15% to 100%), indicating some families are vulnerable once strategy is known.
Experimental setup
Datasets: Next-Gen CAPTCHA procedurally generated instances
Benchmarks: Next-Gen CAPTCHA benchmark
Metrics: Pass@1 cost (USD) completion time Spearman correlation with trajectory metrics
Baselines: Browser-Use CrewAI Claude Cowork
Builds on: OpenCaptchaWorld Browser-Use Playwright
Closest papers by meaning
- Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents (NeurIPS 2025 · similarity 0.90)
- CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training (ICML 2026 · similarity 0.89)
- MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks (AAAI 2026 · similarity 0.88)
- Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiation (ICLR 2026 · similarity 0.86)
- REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites (NeurIPS 2025 · similarity 0.84)
- SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safety (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.