Research map / Vision-language and multimodal / Human and embodied modeling
HENASY: Learning to Assemble Scene-Entities for Interpretable Egocentric Video-Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HENASY, a dual-encoder egocentric video-language model that assembles dynamic scene entities via spatiotemporal token grouping and trains with multi-grained contrastive losses (video-narration, noun-entity, verb-entity) plus a box projection loss, yielding interpretable visual grounding and competitive zero-shot results.
Problem
Video-language models for egocentric video rely on instance-level video-text alignment, which limits fine-grained entity/relationship modeling and offers little interpretability of visual reasoning.
Approach
A hierarchical local entity encoder bootstrapped from early layers of a pretrained TimeSformer groups patches into temporally consistent scene entities via a temporal-aware grouping mechanism; an entity-aware decoder fuses entities with global features, and noun-entity, verb-entity, and projection losses supervise entity-level alignment, enabling saliency-map grounding.
Key results
- Zero-shot EK100-CLS top-1 accuracy of 19.5% vs 16.4% for LaViLa-level baseline in the table, a 3.1-point gain over the re-run HelpingHands (16.4 -> 19.5 reported as 3.1% improvement by authors).
- EgoMCQ inter/intra accuracy of 94.1/61.3 vs HelpingHands* 93.2/58.8.
- Visual grounding mIoU of 41.06% vs 22.73% for HelpingHands on 200 self-labeled Ego4D videos; inference 1.02 s vs 2.87 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the entity-aware decoder instead of average-pooling entity tokens (ablation) | HENASY · EgoMCQ | Inter accuracy: 87.6 -> 94.1 | Helped |
| Add the entity-aware decoder instead of average-pooling entity tokens (ablation) | HENASY · EK100-MIR | Average mAP: 18.8 -> 31.3 | Helped |
| Replace self-attention decoder with the entity-aware decoder (ablation) | HENASY · EgoMCQ | Inter accuracy: 93.3 -> 94.1 | Helped |
| Add the bootstrapping stage (ablation) | HENASY · EgoMCQ | Inter accuracy: 92.6 -> 94.1 | Helped |
| Add noun-entity contrastive loss (ablation) | HENASY · EgoMCQ | Inter accuracy: 93.6 -> 94.1 | Helped |
| Add verb-entities contrastive loss (ablation) | HENASY · EgoMCQ | Inter accuracy: 93.7 -> 94.1 | Helped |
| Add projection loss (ablation) | HENASY · EgoMCQ | Inter accuracy: 93.2 -> 94.1 | Helped |
| Add noun-entity, verb-entities, and projection losses to EgoNCE (ablation) | HENASY · EgoMCQ | Inter accuracy: 93.4 -> 94.1 | Helped |
| Use HENASY rather than HelpingHands for visual grounding | egocentric video grounding · Ego4D visual grounding | mIoU: 22.73% -> 41.06% | Helped |
| Use HENASY rather than HelpingHands* zero-shot transfer | egocentric video models · EgoMCQ | Inter accuracy: 93.2 -> 94.1 | Helped |
| Use HENASY rather than HelpingHands* as feature extractor | episodic-memory models · EgoNLQ | mIoU@0.5: 20.4 -> 21.5 | Helped |
Surprising findings
- Entity-aware decoder replacing average pooling drops EK100-CLS top-1 from 19.5 to 6.7 in the ablation, showing the decoder is critical.
Limitations
- Does not explicitly model interactions between scene entities (no scene graph)
- Evaluated only with ViT-based encoders; not extended to generative tasks or long-form video
- Visual grounding quantitative evaluation uses only 200 self-labeled videos
Experimental setup
Datasets: EgoClip Ego4D EpicKitchens-100 EGTEA
Benchmarks: EgoMCQ EK100-MIR EK100-CLS EGTEA EgoNLQ EgoMQ
Metrics: accuracy mAP nDCG Top-1 Top-5 mIoU Recall@K
Baselines: EgoVLP EgoVLPv2 LaViLa HelpingHands SlowFast VSLNet VSGN
Builds on: EgoVLP LaViLa HelpingHands GroupViT TimeSformer
Closest papers by meaning
- VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment (CVPR 2025 · similarity 0.88)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.88)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.87)
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (CVPR 2024 · similarity 0.87)
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.86)
- ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.