Research map / Vision-language and multimodal / Multimodal LLMs

MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MCA-Bench, a multimodal CAPTCHA benchmark of 20 task types where LoRA-fine-tuned Qwen2.5-VL-7B cracking agents and zero-shot VLMs are evaluated by pass rate against human performance.

Problem

Existing CAPTCHA schemes span many modalities (distorted text, image puzzles, clicks, sliders, logic questions), but there is no unified, large-scale multimodal benchmark for assessing their robustness against VLM-based attacks.

Approach

Integrates heterogeneous CAPTCHA types into one evaluation protocol with a shared Qwen2.5-VL-7B backbone plus per-task LoRA adapters acting as cracking agents, reporting pass rate (Pass@k) and comparing against zero-shot VLMs and humans.

Key results

• LoRA-fine-tuned Qwen2.5-VL-7B reaches about 96-99% pass rate on simple tasks such as hollow-pattern (0.995) and text-based arithmetic (0.985), per the reported tables.
• Zero-shot Qwen2.5-VL-7B scores 0.015 on 3x3 jigsaw-swap and 0.025 on arithmetic selection, while the fine-tuned version reaches 0.025-0.035 on arithmetic selection and 0.700 on arithmetic characters.
• The paper reports that VLMs fall to as low as 2.5% on complex tasks requiring physical interaction or multi-step reasoning, though the tables show inconsistent figures (e.g., sliding block 0.365 fine-tuned).

Surprising findings

• Zero-shot ChatGPT-4o, Seed1.5-VL and Gemini2.5-Pro score near 0 on several tasks such as 3x3 jigsaw-swap, while humans score around 0.98.
• Reported tables and text are inconsistent (e.g., 20 tasks vs. 'CAPTCHA-Bench' naming, differing per-task numbers between Table 1 and the main results), which weakens confidence in the headline figures.

Limitations

• Interactive tasks rely on behavior cloning from a limited set of human demonstrations; new metrics (Center Deviation Error, etc.) are described but not clearly reported.
• Reported numbers are internally inconsistent across text, tables, and naming (e.g., 2.5% claim vs. tables), limiting reliability.
• Images resized to 224x224 may not reflect real deployment conditions.

Experimental setup

Datasets: MCA-Bench training set (self-constructed) Flickr scenic photos 64-class animal dataset Google reCAPTCHA v2 challenges

Benchmarks: MCA-Bench (20 CAPTCHA task types)

Metrics: pass rate Pass@k

Baselines: ChatGPT-4o Seed1.5-VL Gemini2.5-Pro Original Qwen2.5-VL-7B-Instruct Human

Builds on: Qwen2.5-VL LoRA

Closest papers by meaning

• Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents (NeurIPS 2025 · similarity 0.89)
• Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense (ICML 2026 · similarity 0.88)
• CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training (ICML 2026 · similarity 0.86)
• MI-CAPTCHA: Enhance the Security of CAPTCHA Using Mooney Images (AAAI 2025 · similarity 0.86)
• Variance-Based Membership Inference Attacks Against Large-Scale Image Captioning Models (CVPR 2025 · similarity 0.86)
• Spatial CAPTCHA: Generatively Benchmarking Spatial Reasoning for Human-Machine Differentiation (ICLR 2026 · similarity 0.85)