Research map / Video / Action recognition
HD-EPIC: A Highly-Detailed Egocentric Video Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HD-EPIC, a 41-hour set of unscripted multi-day kitchen egocentric videos with dense, 3D-grounded annotations (recipes, nutrition, actions, object motion, gaze, audio) and a 26K-question VQA benchmark on which Gemini Pro reaches 37.6%.
Problem
Existing egocentric video datasets are either sparsely annotated, synthetic, or collected in controlled settings, lacking dense, interconnected, 3D-grounded labels on unscripted real-world footage for evaluating video perception models.
Approach
Collect multi-day unscripted kitchen recordings with Project Aria glasses and layer many interconnected annotation types (recipe prep/step pairs, ingredient nutrition, dense what/how/why narrations, digital twin fixtures, 3D-lifted object masks, gaze priming) into one validation benchmark, with VQA prototypes that use hard negatives sampled from annotations rather than LLM-generated distractors.
Key results
- Gemini Pro achieves the best average VQA accuracy of 37.6% on the 26K-question benchmark, versus 90.3% for a sample human baseline.
- Language-only models (Llama 3.2 90B, Gemini Pro blind) reach only 26.5% and 26.7%, close to the random baseline.
- Best action recognition on HD-EPIC reaches only 51% verbs, 37% nouns and 24% actions, and audio models drop roughly 25-28 points versus EPIC-Sounds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gemini Pro instead of LLaVA-Video | HD-EPIC VQA · HD-EPIC VQA | Avg. accuracy (%): 32.4 -> 37.6 | Helped |
| Use Gemini Pro instead of Llama 3.2 | HD-EPIC VQA · HD-EPIC VQA | Avg. accuracy (%): 26.5 -> 37.6 | Helped |
| Use video-language input instead of language-only input | HD-EPIC VQA · HD-EPIC VQA | Action accuracy (%): 23.3 -> 27.2 | Helped |
| Use TIM with audio and video instead of video only | TIM · HD-EPIC action recognition | Action accuracy (%): 23.9 -> 23.4 | Hurt |
| Use TIM with audio and video instead of audio only | TIM · HD-EPIC sound recognition | Top-1 accuracy (%): 26.9 -> 31.9 | Helped |
| Use SAM2 instead of Cutie | HD-EPIC long-term VOS · HD-EPIC long-term VOS hands | J &F: 77.2 -> 89.1 | Helped |
| Use SAM2 instead of Cutie | HD-EPIC long-term VOS · HD-EPIC long-term VOS objects | J &F: 34.6 -> 27.0 | Hurt |
Surprising findings
- Language-only models score close to chance on the VQA benchmark, showing questions cannot be solved blind.
- SAM2 outperforms Cutie on hands but underperforms it on objects in long-term segmentation.
Limitations
- Validation/test-only benchmark: the dataset is small (41 hours, 9 kitchens) relative to large-scale training corpora.
- Current VLMs perform poorly, especially on questions over a minute long and on nutrition/weights reading.
Experimental setup
Datasets: HD-EPIC EPIC-KITCHENS-100 EPIC-Sounds
Benchmarks: HD-EPIC VQA EPIC-KITCHENS-100 action recognition EPIC-Sounds HD-EPIC long-term VOS
Metrics: accuracy top-1 top-5 mean class accuracy mAP mAUC jaccard index contour accuracy
Baselines: Llama 3.2 90B VideoLlama 2 LongVA LLaVA-Video SlowFast TIM Cutie SAM2
Builds on: EPIC-KITCHENS-100 Ego4D Project Aria SAM2
Closest papers by meaning
- Exploring Domain Incremental Video Highlights Detection with the LiveFood Benchmark (AAAI 2024 · similarity 0.84)
- Understanding Multi-Task Activities from Single-Task Videos (CVPR 2025 · similarity 0.83)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.83)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.83)
- CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities (NeurIPS 2024 · similarity 0.82)
- HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.