Research map / Vision-language and multimodal / Multimodal LLMs

Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiation

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Spatial CAPTCHA, a procedurally generated, invariant-specified CAPTCHA framework built on spatial reasoning tasks, and Spatial-CAPTCHA-Bench (1050 instances), showing humans far outperform 10 MLLMs (best o4-mini at 31.0% pass@1).

Problem

Conventional CAPTCHAs based on text recognition or 2D image understanding are increasingly solved by multimodal LLMs, so a human-machine verification method is needed that exploits spatial reasoning gaps between humans and MLLMs.

Approach

Generate CAPTCHA items from machine-checkable task manifests tied to cognitive spatial invariants (perspective-taking, mental rotation, multi-step visualization, reference frames), with procedural scene generation, distractor synthesis, automated validators, and a difficulty map calibrated on human response times and success rates.

Key results

• Best MLLM o4-mini reaches 31.0% pass@1 on Spatial-CAPTCHA-Bench versus 89.5% for human-simple baseline.
• Gemini-2.5-Pro scores 29.0% on Spatial-CAPTCHA-Bench but 55.3% on reCAPTCHA-Bench.
• Models are poorly calibrated: all models fall below the identity line in k-of-k vs pass@k reliability analysis.

Surprising findings

• Human accuracy on Spatial-CAPTCHA-Bench (Tiny) is slightly higher than on reCAPTCHA-Bench, while several frontier MLLMs score far lower on the new benchmark than on reCAPTCHA.
• Slower models are not more accurate; latency is only weakly coupled with accuracy.

Limitations

• Human evaluation uses a small 70-item Tiny subset with 60 raters and a 30-second limit.
• Models are evaluated zero-shot without chain-of-thought, and the difficulty map relies on pilot human data.
• Only 7 task formulations are currently included; the benchmark is offline and not yet tested as a deployed CAPTCHA against real attackers.

Experimental setup

Datasets: reCAPTCHA-Bench

Benchmarks: Spatial-CAPTCHA-Bench reCAPTCHA-Bench

Metrics: pass@1 pass@k (k=3) k-of-k reliability per-ability pass@1 human pass rate

Baselines: chatgpt-4o-latest gemini-2.5-pro o4-mini gemini-2.5-flash claude-sonnet-4 claude-opus-4 qwen2.5-vl-72b-instruct llama-4-maverick mistral-medium-3 phi-4-multimodal-instruct

Builds on: Google reCAPTCHA

Closest papers by meaning

• Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents (NeurIPS 2025 · similarity 0.90)
• CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training (ICML 2026 · similarity 0.88)
• Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.87)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.87)
• SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.87)
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.87)