Research map / Vision-language and multimodal / Visual question answering
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IndEgo, a ~197-hour egocentric (plus ~97-hour exocentric) multimodal dataset of industrial tasks with a strong collaborative-work focus, and benchmarks VLMs on mistake detection, reasoning QA and collaborative task understanding.
Problem
Egocentric vision datasets rarely cover real industrial settings, especially collaborative, physically and cognitively demanding work, so assistants for industrial workers lack suitable data and benchmarks.
Approach
A multimodal ego+exo dataset recorded with Project Aria glasses in an industrial facility, covering assembly/disassembly, logistics, inspection/repair, woodworking and others, with two-person collaborative recordings, eye gaze, narration, audio and motion, plus benchmarks for mistake detection, QA and collaborative task understanding.
Key results
- On mistake detection, zero-shot Gemini 2.0 Flash Thinking reaches 40.9 F1 versus 23.3-24.2 for open VLMs, and the best finetuned MLP reaches 39.5 F1.
- On reasoning QA over singular actions, the best VLM (Gemini 2.0 Flash Thinking) reaches 64.1% accuracy versus 90.0% for humans.
- Joint ego+exo views give the best mistake detection F1 across models (e.g., 0.44 for GFT zero-shot vs 0.43 ego-only).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use joint ego and exo views instead of either view alone | Mistake detection · 10-task view comparison | F1: 0.43 -> 0.44; 0.39 -> 0.44 | Helped |
| Add audio and gaze to RGB (ablation) | VideoLLaMA3 · Mistake Detection (10 tasks) | F1: 0.27 -> 0.30 | Helped |
| Add audio to RGB (ablation) | InternVL-2.5 · Mistake Detection (10 tasks) | F1: 0.30 -> 0.28 | Hurt |
| Add gaze to RGB (ablation) | InternVL-2.5 · Mistake Detection (10 tasks) | F1: 0.30 -> 0.29 | Hurt |
| Add audio and gaze to RGB (ablation) | Gemini 2.0 Flash Thinking Experimental · Mistake Detection (10 tasks) | F1: 0.38 -> 0.42 | Helped |
| Reduce video frame rate from 10 FPS to 5 FPS (ablation) | Gemini 2.0 Flash Thinking Experimental · Mistake Detection (10 tasks) | F1: 3% drop | Hurt |
| Use the exocentric view instead of the egocentric view | Gemini 2.0 Flash Thinking Experimental · Mistake Detection (first 15 tasks) | F1: 41.1 -> 41.6 | Helped |
| Use audio narration transcripts only | Long-task reasoning QA · Long-task QA | Accuracy: 67.3% | No clear effect |
| Provide the action label to Mistral-Large2 | Mistral-Large2 · Reasoning-based QA (singular actions) | Accuracy: 92.3 (Tm); 51.4 (Si); 42.8 (Re); 78.3 (A); 61.4 (overall) | Mixed |
| Use a Transformer classifier instead of an MLP classifier | VideoLLaMA3 Mistake Detection classifier · Mistake Detection | F1: 39.5 -> 33.9 | Hurt |
| Use early mistake detection with only the first 50% of segment frames | Mistake Detection classifier · Early Mistake Detection | F1: 39.5 -> 30.7 | Hurt |
| Use joint ego and exo views | Mistake detection zero-shot models · 10-task view comparison | F1: GFT: 0.43 -> 0.44; VL3: 0.33 -> 0.37; IVL2.5: 0.29 -> 0.33 | Mixed |
Surprising findings
- Reducing frame rate from 10 to 5 FPS caused only about a 3% F1 drop in mistake detection.
- Removing audio in noisy environments can slightly improve mistake detection performance for some tasks.
Limitations
- Only 20 participants (15 male, 5 female), limiting demographic coverage.
- Data collected at a single industrial research facility, not in true production sites due to privacy and IP constraints.
- Inter-annotator agreement for fine-grained actions is low (Krippendorff's alpha 0.25 strict, 0.54 grouped).
Experimental setup
Datasets: IndEgo
Benchmarks: IndEgo Mistake Detection IndEgo Reasoning-based QA IndEgo collaborative task understanding
Metrics: F1 precision recall accuracy Krippendorff's alpha
Baselines: VideoLLaMA3 InternVL-2.5 Qwen2.5-VL-7B Gemini 2.0 Flash Thinking Mistral-Large2
Builds on: Project Aria EPIC-KITCHENS Ego4D Ego-Exo4D
Closest papers by meaning
- EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World (CVPR 2024 · similarity 0.86)
- ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction (CVPR 2025 · similarity 0.85)
- LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks (CVPR 2026 · similarity 0.85)
- EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World? (NeurIPS 2025 · similarity 0.84)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.84)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.