Research map / Video / Video-language understanding
Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds HOD, an LLM-based pipeline that rewrites ego video narrations with hand-object trajectory and contact info, and EgoVideo, a ViT video-language model with a lightweight motion adapter co-trained at two frame rates.
Problem
Egocentric video-language representation learning relies on highly condensed narrations that omit fine-grained hand-object motion and contact dynamics.
Approach
Generate hand-object dynamics captions by feeding detector-derived hand/object box trajectories and contact states into an LLM (Yi-34B) to rephrase original narrations, then pretrain a dual-branch ViT with a separable spatial-temporal motion adapter at a higher frame rate, co-trained with the backbone at a lower frame rate.
Key results
- EgoVideo-B zero-shot EK-100 MIR mAP 36.5 vs EMBED-B 36.0 and LaViLa-B 30.9; EgoVideo-G reaches 47.1 mAP.
- EgoVideo-G zero-shot EGTEA mean accuracy 58.0 vs prior results; EgoVideo-B EGTEA top-1 51.0.
- Ego4D-HOD pretraining raises EgoVideo-B EK-100 MIR mAP from 31.1 (EgoClip) to 34.4 and EGTEA top-1 from 36.0 to 47.9 under matched settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace EgoClip with Ego4D-HOD pretraining data (ablation) | EgoVideo-B · EGTEA (zero-shot) | Top1-acc.: 36.0 -> 47.9 | Helped |
| Add How2-HOD to EgoClip pretraining data (ablation) | EgoVideo-B · EK-100 MIR (zero-shot) | mAP: 31.1 -> 35.5 | Helped |
| Use Ego4D-HOD and How2-HOD instead of EgoClip alone (ablation) | EgoVideo-B · EGTEA (zero-shot) | Top1-acc.: 36.0 -> 51.0 | Helped |
| Add spatial convolution to the temporal-only adapter (ablation) | EgoVideo motion adapter · EK-100 MIR | mAP: 35.9 -> 36.5 | Helped |
| Increase sampled frame count from 4 to 16 (ablation) | EgoVideo · EK-100 MIR | mAP: 34.2% to 36.5% | Helped |
| Increase sampled frame count beyond 16 to 32 (ablation) | EgoVideo · EK-100 MIR | mAP | No clear effect |
| Use Ego4D-HOD instead of EgoClip at fixed data size (ablation) | EgoVideo-B · EK-100 MIR (zero-shot) | mAP: 31.1 -> 34.4 | Helped |
| Use Ego4D-HOD instead of EgoClip at fixed data size (ablation) | EgoVideo-L · EK-100 MIR (zero-shot) | mAP: 35.3 -> 38.3 | Helped |
| Use Ego4D-HOD instead of EgoClip at fixed data size (ablation) | EgoVideo-G · EK-100 MIR (zero-shot) | mAP: 42.1 -> 44.8 | Helped |
| Train with EgoVideo mixed 4- and 16-frame strategy instead of AVION-16f (ablation) | AVION/EgoVideo training comparison · EGTEA | Top1-acc.: 47.4 -> 51.0 | Helped |
Surprising findings
- Motion adapter with a higher frame rate gives gains over ST-adapter and standard adapter, but the paper notes that increasing adapter downsampling ratio keeps improving performance.
- Helping Hands and MPI-based robot results show EgoVideo is comparable to MPI rather than uniformly superior on Franka Kitchen.
Limitations
- Performance depends on hand-object detector quality and LLM rephrasing, which can accumulate errors.
- Narration quality measured only with GPT-4o scores and METEOR/CIDEr; HOD has lower METEOR than LaViLa-Narrator.
Experimental setup
Datasets: Ego4D How2-Interlink7M EK-100 EGTEA GTEA HOI4D Franka Kitchen
Benchmarks: EK-100 MIR EK-100 CLS EGTEA EgoMCQ EgoNLQ EgoMQ GTEA HOI4D Franka Kitchen
Metrics: mAP nDCG top-1 accuracy top-5 accuracy mean accuracy R1@0.5 F1@50 Edit success rate GPT-Score
Baselines: EgoVLPv2 LaViLa AVION EMBED Helping Hands MVP Voltron MPI ASFormer
Builds on: EgoVLP LaViLa AVION CLIP 100DOH
Closest papers by meaning
- Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions? (ICLR 2025 · similarity 0.91)
- Joint Hand Motion and Interaction Hotspots Prediction From Egocentric Videos (CVPR 2022 · similarity 0.90)
- HandWorld: Hand-Centric Unified Video Action Generation (CVPR 2026 · similarity 0.90)
- MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation (NeurIPS 2025 · similarity 0.89)
- Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.88)
- The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.