Research map / Vision-language and multimodal / Multimodal LLMs
EgoBlind: Towards Egocentric Visual Assistance for the Blind
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoBlind, an egocentric VideoQA dataset of 1,392 videos filmed by blind people with 5,311 blind-posed or blind-verified questions, and benchmarks 16 MLLMs, finding best models near 60% vs 87.4% human accuracy.
Problem
Existing VQA and egocentric video datasets do not reflect the real assistive needs of blind people, since they use third-person or sighted-simulated footage and questions; MLLMs are not evaluated for live egocentric visual assistance for the blind.
Approach
A VideoQA benchmark whose videos are captured by blind content creators and whose questions are extracted from them, generated by GPT-4o acting as a blind user, or written by blind annotators, with timestamp-restricted online QA, multiple reference answers, and GPT-based scoring.
Key results
- Best closed-source model GPT-4o reaches 59.3% overall accuracy versus 87.4% human performance.
- Open-source InternVL2.5-26B reaches 55.0%, and LLaVA-Video 51.5%, showing general egocentric VQA strength does not transfer.
- LoRA finetuning on the EgoBlind training split improves Qwen2.5-VL, LLaVA-OV and InternVL2.5-8B by roughly 2.9 to 5.6 points under the user-aware split, though gaps to humans remain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| instruction-tune models on EgoBlind training data (ablation) | Qwen2.5-VL, LLaVA-OV, and InternVL2.5-8B · EgoBlind | accuracy: 45.5 -> 50.2; 54.5 -> 57.4; 53.5 -> 58.1 | Helped |
| use a user-aware rather than user-agnostic data split (ablation) | Qwen2.5-VL, LLaVA-OV, and InternVL2.5-8B · EgoBlind | accuracy: 45.5 -> 47.7; 54.5 -> 56.4; 53.5 -> 56.2 | Helped |
| use category-specific prompts (ablation) | GPT-4o · EgoBlind test subset | accuracy: 59.4 -> 62.2 | Helped |
| use category-specific prompts (ablation) | LLaVA-OV and InternVL2.5-26B · EgoBlind test subset | accuracy: 54.2 -> 52.2; 54.6 -> 53.1 | Hurt |
| remove subtitles (ablation) | Qwen2.5-VL, LLaVA-OV, InternVL2.5-26B, and GPT-4o · EgoBlind test subset | accuracy: 44.5 -> 43.2; 54.2 -> 53.7; 54.6 -> 53.8; 59.4 -> 57.6 | Hurt |
| add extracted scene text to the prompt (ablation) | Qwen2.5-VL, LLaVA-OV, InternVL2.5-26B, and GPT-4o · EgoBlind test subset | accuracy: 44.5 -> 44.8; 54.2 -> 54.1; 54.6 -> 54.2; 59.4 -> 56.7 | Mixed |
| translate questions into Chinese (ablation) | Qwen2.5-VL, LLaVA-OV, InternVL2.5-26B, and GPT-4o · EgoBlind test subset | accuracy: 44.5 -> 41.5; 54.2 -> 51.4; 54.6 -> 53.1; 59.4 -> 55.9 | Hurt |
| replace multi-frame video input with a single frame at the question timestamp (ablation) | InternVL2.5-26B, Gemini 2.5 Flash, and GPT-4o · EgoBlind | accuracy | Hurt |
| replace the blind-specific prompt with a normal VQA prompt (ablation) | InternVL2.5-26B, Gemini 2.5 Flash, and GPT-4o · EgoBlind | accuracy | Hurt |
| instruction-tune on EgoBlind training data (ablation) | models evaluated on Tool Use · EgoBlind Tool Use | accuracy | Hurt |
| provide more video frames (ablation) | InternVL models · EgoBlind | accuracy | No clear effect |
| increase the model size and use stronger LLMs | MLLMs evaluated on EgoBlind · EgoBlind | accuracy | Helped |
Surprising findings
- Gemini 1.5 Flash surpasses human performance on tool-use questions (72.5 vs 70.4), while closed-source models do not uniformly beat open-source ones.
- Replacing the video with a single frame at the question timestamp only slightly lowers accuracy, and blind-specific prompting helps GPT-4o but not open-source models.
Limitations
- Data limited to videos from mainland China, mostly Chinese platforms and questions translated from Chinese.
- Training set is small compared with sighted-person video QA datasets.
- Only a snapshot of MLLMs was evaluated and newer models may be missing.
Experimental setup
Datasets: EgoBlind
Benchmarks: EgoBlind
Metrics: accuracy (GPT-judged) Cohen's kappa
Baselines: Human GPT-4o Gemini 2.5 Flash Gemini 1.5 Flash Gemini 2.0 Flash InternVL2.5-26B LLaVA-OV Qwen2.5-VL LLaVA-Video VILA1.5
Builds on: VizWiz Ego4D EgoTaskQA LoRA
Closest papers by meaning
- In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting (NeurIPS 2025 · similarity 0.87)
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.87)
- EgoSound: Benchmarking Sound Understanding in Egocentric Videos (CVPR 2026 · similarity 0.87)
- EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (CVPR 2024 · similarity 0.86)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.86)
- EgoLife: Towards Egocentric Life Assistant (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.