Research map / Video / Video-language understanding
EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoTextVQA, a 1.5K ego-view video, 7K question benchmark for scene-text-aware egocentric video QA in driving and housekeeping, and benchmarks 10 MLLMs, finding the best (Gemini 1.5 Pro) reaches about 33-34% accuracy.
Problem
Existing scene-text VQA benchmarks use well-focused images or questions that point to text, and egocentric VideoQA benchmarks ignore scene text, so they do not reflect real assistive QA needs where users ask questions without pinpointing text in dynamic first-person video.
Approach
Builds a benchmark of egocentric videos with scene-text-rich content and user-intent questions asked at timestamps, restricting models to video content before each question timestamp to simulate live streaming QA, and uses GPT-4o generation with heavy human curation; then probes heuristics such as frame selection, resolution, and OCR/super-resolution inputs.
Key results
- Gemini 1.5 Pro is the best model, reaching 33.4% accuracy on EgoTextVQA-Outdoor and 34.4% on EgoTextVQA-Indoor, while humans score 43.1% and 27.7%.
- Higher input resolution and scene-text super-resolution improve results; e.g., Gemini 1.5 Pro with SR at 1.25x reaches 38.1% on Outdoor.
- Adding OCR scene-text input to video raises GPT-4o outdoor accuracy from 30.3% to 52.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add scene-text input to video and question (ablation) | GPT-4o · EgoTextVQA-Outdoor | accuracy: 30.3 -> 52.9 | Helped |
| Add scene-text input to video and question (ablation) | Gemini 1.5 Pro · EgoTextVQA-Indoor | accuracy: 34.4 -> 39.5 | Helped |
| Use three QA-generation frames instead of uniformly sampled video frames (ablation) | Qwen2-VL · EgoTextVQA-Indoor | accuracy: 23.3 -> 30.7 | Helped |
| Use three QA-generation frames instead of uniformly sampled video frames (ablation) | Gemini 1.5 Pro · EgoTextVQA-Outdoor | accuracy: 33.4 -> 26.3 | Hurt |
| Use a single frame at the question timestamp instead of video (ablation) | GPT-4o · EgoTextVQA-Indoor | accuracy: 28.3 -> 14.8 | Hurt |
| Add scene-text super-resolution at 1.50× scale (ablation) | Qwen2-VL · EgoTextVQA-Outdoor | accuracy: 28.2 -> 34.1 | Helped |
| Add scene-text super-resolution at 1.50× scale (ablation) | Qwen2-VL · EgoTextVQA-Indoor | accuracy: 23.3 -> 22.3 | Hurt |
| Add video to question and scene-text input (ablation) | Qwen2-VL · EgoTextVQA-Outdoor | accuracy: 15.4 -> 39.9 | Helped |
| Add video to question and scene-text input (ablation) | LLaVA-NeXT-Video · EgoTextVQA-Indoor | accuracy: 15.6 -> 30.8 | Helped |
Surprising findings
- Humans perform worse than the closed-source models on EgoTextVQA-Indoor (27.7% vs. 34.4% for Gemini 1.5 Pro), attributed to scene-text recognition difficulty and external knowledge needs.
- Using a single frame at the question timestamp hurts Gemini 1.5 Pro and Qwen2-VL on some subsets, while Qwen2-VL improves with a single frame on the outdoor set.
Limitations
- Human answer diversity lowers human scores, so ground-truth answers are not exhaustive.
- Automatic QA generation required heavy manual correction; about 10% of generated questions were answerable blindly and removed.
- All evaluated models perform near 33% so the benchmark is far from saturated, but results depend on GPT-4o mini judging.
Experimental setup
Datasets: RoadTextVQA EgoSchema
Benchmarks: EgoTextVQA-Outdoor EgoTextVQA-Indoor
Metrics: accuracy score (0-5)
Baselines: GPT-4o Gemini 1.5 Flash Gemini 1.5 Pro Qwen2-VL InternVL2 LLaVA-NeXT-Video VILA1.5 MiniCPM-V 2.6 ShareGPT4Video CogVLM2-Video
Builds on: RoadTextVQA EgoSchema TextVQA DiffTSR
Closest papers by meaning
- Grounded Question-Answering in Long Egocentric Videos (CVPR 2024 · similarity 0.89)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.89)
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.88)
- Multi-Factor Adaptive Vision Selection for Egocentric Video Question Answering (ICML 2024 · similarity 0.88)
- NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario (AAAI 2024 · similarity 0.87)
- Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.