Research map / Vision-language and multimodal / Visual question answering
WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WearVQA, a 2,520-item egocentric VQA benchmark for smart-glasses assistants with quality issues and reasoning tasks, evaluated with an LLM-as-judge (GPT-4o) across 10 MLLMs.
Problem
Existing VQA benchmarks use high-quality third-person images and do not reflect wearable egocentric conditions (occlusion, low light, unzoomed, blur) or wearable-relevant questions.
Approach
A benchmark of egocentric images captured with smart glasses, annotated with image-based, short-form, unambiguous questions across 10 cognitive task types and 6 wearable-specific image quality issues, scored by a GPT-4o LLM judge checked against human labels (96% accuracy).
Key results
- GPT-4o achieved the top QA accuracy of 51.5% on WearVQA public test set (high-quality 58.5%, low-quality 45.5%), with GPT-4.1 at 53%.
- Open-source models ranged from 21.3% to 45.1% accuracy on the benchmark; Qwen2.5-VL-72B was the best open-source model at 45.1%.
- The LLM judge reached 96% accuracy against human annotations on 10% of responses, with precision 98.2% and recall 95.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate on low-quality rather than high-quality images | Phi-4-mini-instruct (3.8B) · WearVQA | accuracy: 26.9 -> 21.3 (-5.6) | Hurt |
| Evaluate on low-quality rather than high-quality images | Molmo-72B-0924 · WearVQA | accuracy: 29.7 -> 22.2 (-7.5) | Hurt |
| Evaluate on low-quality rather than high-quality images | Pixtral-12B-2409 · WearVQA | accuracy: 29.5 -> 22.7 (-6.8) | Hurt |
| Evaluate on low-quality rather than high-quality images | Llama-3.2v-90B-Vision · WearVQA | accuracy: 41.6 -> 33.8 (-7.8) | Hurt |
| Evaluate on low-quality rather than high-quality images | Llama-4-Scout-17B-16E-Instruct · WearVQA | accuracy: 45.6 -> 38.0 (-7.6) | Hurt |
| Evaluate on low-quality rather than high-quality images | Llama-4-Maverick-17B-128E-Instruct · WearVQA | accuracy: 45.9 -> 39.3 (-6.6) | Hurt |
| Evaluate on low-quality rather than high-quality images | Qwen2.5-VL-72B-Instruct · WearVQA | accuracy: 51.1 -> 39.9 (-11.2) | Hurt |
| Evaluate on low-quality rather than high-quality images | Claude-3.7-sonnet (175B) · WearVQA | accuracy: 42.6 -> 26.4 (-16.2) | Hurt |
| Evaluate on low-quality rather than high-quality images | Gemini-1.5-Pro (200B) · WearVQA | accuracy: 50.4 -> 41.0 (-9.4) | Hurt |
| Evaluate on low-quality rather than high-quality images | GPT-4o (200B) · WearVQA | accuracy: 58.5 -> 45.5 (-13.0) | Hurt |
| Evaluate on mid-resolution rather than low-resolution images | Llama-4-Maverick · WearVQA high-resolution subset, overall | accuracy: 28.8 -> 32.7 (+3.9) | Helped |
| Evaluate on mid-resolution rather than low-resolution images | Qwen2.5-VL · WearVQA high-resolution subset, overall | accuracy: 38.0 -> 36.1 ( -1.9) | Hurt |
Surprising findings
- Low-light images had minimal impact on QA quality, with top models reaching 58% on them, which the authors attribute to focused illumination of the main subject.
- Quality drops were not consistent with resolution: lower resolution decreased accuracy for some models (Llama-4, Gemini-1.5) but increased it for Qwen2.5 and GPT-4o.
Limitations
- Only English-language queries are covered.
- Static image-only design does not capture temporal dynamics or video reasoning.
- Questions must be answerable without external knowledge, excluding RAG-style queries.
Experimental setup
Benchmarks: WearVQA
Metrics: QA accuracy
Baselines: GPT-4o Gemini-1.5-Pro Claude-3.7-sonnet Qwen2.5-VL-72B-Instruct Llama-4-Maverick-17B-128E-Instruct Llama-4-Scout-17B-16E-Instruct Llama-3.2v-90B-Vision Phi-4-mini-instruct
Builds on: VQA v2 OK-VQA MMBench MMMU MMVet
Closest papers by meaning
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.88)
- EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (CVPR 2024 · similarity 0.88)
- NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario (AAAI 2024 · similarity 0.87)
- EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering (CVPR 2025 · similarity 0.87)
- Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents (NeurIPS 2025 · similarity 0.86)
- OpenEQA: Embodied Question Answering in the Era of Foundation Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.