Research map / Vision-language and multimodal / Multimodal LLMs

Towards Scalable Web Accessibility Audit with MLLMs as Copilots

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AAA, a human-AI web accessibility audit framework aligned with WCAG-EM, combining GRASP (GNN-based multimodal page sampling) and MaC (MLLM copilots), plus four new datasets (TPS, APR, CCT, CPE).

Problem

Web accessibility audits following WCAG-EM are labor-intensive and unscalable; existing page sampling relies on text-only statistics and existing MLLM uses cover only evaluation/remediation, not the full audit lifecycle.

Approach

Fuse BERT text embeddings, ViT screenshot embeddings and hyperlink graph via a GNN, refine the graph with representativeness-guided edge pruning/recovery, and sample cluster centroids (GRASP); use MLLMs as assistants/auditors for structured page recognition, element localization, and cognitive CAPTCHA accessibility judgments (MaC), with fine-tuned small models.

Key results

• On TPS, GRASP_IGNN reaches the lowest sampled inter-cluster layout similarity (44.31 vs 54.17-56.66 for SDC variants) and highest D_intra-inter (14.94 layout).
• On CCT, fine-tuned InternVL2-8B achieves 100% CAPTCHA existence recall and 45.58 macro F1, outperforming GPT-4o (29.16 macro F1).
• On CPE/APR, GPT-4o leads on several element types (e.g., search recall 98.80%), while GPT-4o-mini and Qwen differ by category, suggesting no single model dominates.

Surprising findings

• A fine-tuned 8B InternVL2 outperforms GPT-4o (200B) on CCT cognitive accessibility judgments.
• Without fine-tuning, small MLLMs often fail to produce structured, parseable outputs.

Limitations

• Small evaluation: APR and CPE use only five websites or 1,199 pages; CCT has 1,985 images.
• Consultant (remediation) role is not implemented; only assistant and auditor roles are evaluated.
• Sampling evaluated with intrinsic embedding-space metrics rather than human auditor validation.

Experimental setup

Datasets: TPS APR CCT CPE

Benchmarks: TPS APR CCT CPE

Metrics: S_sampled D_intra-inter accuracy precision recall macro F1 violation accuracy

Baselines: SDC_content SDC_struc_cont SDC_structure SDC_tags SDC_tree GRASP_GCN GPT-4o GPT-4o-mini Qwen2.5-VL-72B MiniCPM-V 8B

Builds on: WCAG-EM Web Structure Derived Clustering (SDC) graph structure learning (HoLe) GCN IGNN

Closest papers by meaning

• Consent in Crisis: The Rapid Decline of the AI Data Commons (NeurIPS 2024 · similarity 0.81)
• Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs (NeurIPS 2024 · similarity 0.81)
• Harnessing Webpage UIs for Text-Rich Visual Understanding (ICLR 2025 · similarity 0.80)
• RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users (AAAI 2026 · similarity 0.80)
• WebVLN: Vision-and-Language Navigation on Websites (AAAI 2024 · similarity 0.80)
• Bridging the Data Provenance Gap Across Text, Speech, and Video (ICLR 2025 · similarity 0.80)