Research map / Video / Video-language understanding
Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EgoHOIBench, a multi-choice test showing egocentric video-language models fail on verb/noun substitutions in HOI captions, and EgoNCE++, an asymmetric contrastive objective with LLM/vocabulary-generated hard negatives (V2T) and noun-clustered positives (T2V) that continues pretraining EgoVLP, EgoVLPv2, and LaViLa.
Problem
Egocentric video-language models (EgoVLMs) are trained on hand-object interaction data but fail to distinguish sentences whose verb or noun is substituted, suggesting weak fine-grained understanding of HOI combinations.
Approach
An asymmetric contrastive objective: video-to-text loss augmented with LLM- or vocabulary-generated HOI verb/noun hard negatives, and text-to-video loss that only treats videos sharing the same noun as positives to preserve the object-centric feature space; applied via LoRA continued pretraining on the visual encoder.
Key results
- On EgoHOIBench, LaViLa++ raises action accuracy from 36.85% to 63.17% (+26.32 points) over LaViLa in zero-shot evaluation.
- On EK-100-MIR zero-shot, LaViLa++ improves average mAP from 30.8 to 32.0 over LaViLa.
- On ActionBench, LaViLa++ reaches 91.18% action accuracy versus 79.89% for LaViLa, close to the 92.0% human score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue pretraining LaViLa with EgoNCE++ | LaViLa · EgoHOIBench | action accuracy: +26.32% | Helped |
| Continue pretraining LaViLa with EgoNCE++ | LaViLa · ActionBench | action accuracy: +11.3% | Helped |
| Continue pretraining LaViLa with EgoNCE++ | LaViLa · EK-100-CLS | action accuracy: +4.53% | Helped |
| Continue pretraining LaViLa with EgoNCE++ | LaViLa · EGTEA | mean accuracy: +3.1% | Helped |
| Use EgoNCE++ for V2T rather than InfoNCE, keeping InfoNCE for T2V (ablation) | EgoVLP · EgoHOIBench | action (%): 30.51 -> 40.31 | Helped |
| Use EgoNCE++ for both V2T and T2V rather than EgoNCE for both (ablation) | EgoVLP · EgoHOIBench | action (%): 30.16 -> 41.63 | Helped |
| Generate hard negatives with LLM rather than rule-based selection (ablation) | EgoVLP · EgoHOIBench | action (%): 32.63 -> 41.63 | Helped |
| Add verb negatives (ablation) | EgoVLP V2T · EgoHOIBench | verb (%): +14.89% | Helped |
| Add noun negatives (ablation) | EgoVLP V2T · EgoHOIBench | noun (%): +0.17% | Helped |
| Group positive videos by nouns only rather than verbs and nouns (ablation) | EgoVLP T2V · EgoHOIBench | action (%): 41.26 -> 41.63 | Helped |
| Fully tune the visual encoder instead of LoRA-tuning it, with text encoder frozen (ablation) | EgoVLP · EgoHOIBench | action accuracy: +2.76% | Helped |
| Fully tune the visual encoder instead of LoRA-tuning it, with text encoder frozen (ablation) | EgoVLP · EK-100-MIR | mAP: -0.3% | Hurt |
Surprising findings
- Fine-tuning the text encoder (word embeddings) improves EgoHOIBench but hurts generalization on EK-100-MIR, indicating overfitting to the pretraining data.
- Noun-based hard negatives gave only a marginal gain (+0.17%) compared with verb negatives (+14.89%) for verb accuracy.
Limitations
- Limited to hand-object interaction scenarios; broader egocentric activities are not covered.
- Object recognition is hard to improve through text supervision alone, given the diversity of object categories and limited negatives.
- Gains on mAP in multi-instance retrieval are mixed; LaViLa++ lowers mAP on fine-tuned EK-100-MIR.
Experimental setup
Datasets: EgoClip-3.8M Ego4D EgoMCQ
Benchmarks: EgoHOIBench EK-100-MIR EK-100-OV EK-100-CLS EGTEA CharadesEgo ActionBench
Metrics: accuracy mAP nDCG mean-acc top-1 acc
Baselines: EgoVLP EgoVLPv2 LaViLa HelpingHands HENASY InternVideo Clip-ViP Singularity
Builds on: EgoNCE InfoNCE EgoVLP LaViLa LoRA
Closest papers by meaning
- Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning (ICLR 2025 · similarity 0.91)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.89)
- Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation (CVPR 2024 · similarity 0.89)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.88)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.87)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.