Research map / Vision-language and multimodal / Cross-modal learning
Reading Recognition in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces reading recognition in the wild, a 100-hour egocentric multimodal Reading in the Wild dataset (Project Aria gaze, RGB, IMU), and a lightweight transformer that fuses gaze, RGB crops and head-pose inputs to detect when a user is reading.
Problem
Detecting when a smart-glasses wearer is reading, in unconstrained real-world settings, as an efficient proxy signal for invoking heavier OCR/VLM models under wearable power and bandwidth constraints.
Approach
A tiny multimodal transformer (137k params) that encodes 2s of 3D eye gaze, a 64px RGB crop around the gaze point, and IMU/VIO head-pose, with modality dropout, trained on a large diverse in-the-wild reading dataset; gaze-only and RGB-only models are also supported.
Key results
- Gaze-only and RGB-only models reach 82.3% and 82.2% accuracy on Seattle test; combining all three modalities gives 86.9% accuracy (+4.6% over best single modality).
- Zero-shot on Columbus, gaze+RGB+IMU reaches 82.9% accuracy and 88.8 F1.
- Training on EGTEA does not generalize well to Seattle (62.9% accuracy), while Seattle-trained model transfers better to EGTEA (87.7% accuracy on EGTEA test).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine gaze, RGB, and IMU instead of using the best single modality | Reading recognition model · Seattle test | accuracy: 82.3% -> 86.9% (+4.6%) | Helped |
| add IMU as a secondary modality to gaze | Reading recognition model · Seattle test | accuracy: +2.6% | Helped |
| add IMU as a secondary modality to RGB | Reading recognition model · Seattle test | accuracy: +1.3% | Helped |
| add IMU to gaze and RGB | Reading recognition model · Columbus zero-shot | accuracy: 82.8 -> 82.9 | Helped |
| rotate gaze for vertical Chinese text | Gaze-only reading model · Chinese cross-language | accuracy: 35.5 -> 85.1 (+49.6) | Helped |
| flip gaze horizontally for right-to-left Arabic text | Gaze-only reading model · Arabic cross-language | accuracy: 21.0 -> 51.5 (+30.5) | Helped |
| train on EGTEA instead of Seattle | Reading recognition model · Seattle test | accuracy: 79.3 -> 62.9 (-16.4) | Hurt |
| replace raw IMU with VIO (ablation) | Gaze + head-pose model · Reading recognition | accuracy: 83.9 -> 84.9 | Helped |
| use time-differentiated 3D point gaze input instead of retina images (ablation) | Gaze encoder · Reading recognition | accuracy: 79.2 -> 82.3 | Helped |
| increase gaze frequency from 6 Hz to 60 Hz (ablation) | Gaze encoder · Reading recognition | accuracy: 79.2 -> 82.3 | Helped |
| increase gaze duration from 1 s to 5 s (ablation) | Gaze encoder · Reading recognition | accuracy: 79.6 -> 85.8 | Helped |
| increase RGB crop from 5° FoV to 14° FoV (ablation) | RGB encoder · Reading recognition | accuracy: 82.2 -> 83.5 | Helped |
Surprising findings
- A 6k-parameter model (S size) performs surprisingly well, reaching 86.3% accuracy when scaled to 34k parameters.
- Rotating gaze at inference boosts Chinese (vertical) text accuracy from 35.5% to 85.1%, and horizontal flipping raises Arabic accuracy from 21.0% to 51.5%.
Limitations
- Fails on atypical reading such as reading while writing or typing, and on non-text content like maps and music sheets.
- Struggles with hard negatives where text is present but not read.
- Accuracy is only about 87%, and fine-grained reading mode and medium classification is difficult.
Experimental setup
Datasets: Reading in the Wild (Seattle subset) Reading in the Wild (Columbus subset) EGTEA Gaze+
Benchmarks: Reading in the Wild Seattle test Reading in the Wild Columbus EGTEA Gaze+
Metrics: accuracy F1 precision at 0.9 recall latency
Baselines: gaze-only model RGB-only model IMU-only model EGTEA-trained model
Builds on: Project Aria EGTEA Gaze+ WhisperX EgoBlur
Closest papers by meaning
- WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarios (NeurIPS 2025 · similarity 0.82)
- Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents (NeurIPS 2025 · similarity 0.82)
- Yo'LLaVA: Your Personalized Language and Vision Assistant (NeurIPS 2024 · similarity 0.81)
- EgoBlind: Towards Egocentric Visual Assistance for the Blind (NeurIPS 2025 · similarity 0.81)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.81)
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.