Research map / Vision-language and multimodal / Human and embodied modeling
Beyond Grounding: Extracting Fine-Grained Event Hierarchies across Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the task of extracting hierarchical and identical event relations between news article text and video, releases the MultiHiEve news video-article dataset, and proposes MASHER, a weakly supervised model trained on pseudo-labels.
Problem
Existing multimodal event work uses grounding, which only tells whether text and video events are identical and cannot capture hierarchical (subevent) relations across modalities at different semantic levels.
Approach
Formulates multimodal event hierarchy extraction (hierarchical and identical relations from text events to video events), builds pseudo-labels by combining text hierarchy detection with CLIP-based video retrieval, and trains MASHER to predict relations directly, enabling discovery of visual-only subevents.
Key results
- MASHER reaches 18.0 average F1 on the validation set versus 11.4 for the strongest baseline (MM Baseline).
- MASHER achieves 22.0 hierarchical F1 on validation, about 4x the recall of MM Baseline on hierarchical relations.
- MASHER recovers 15.92% of visual-only subevents versus 2.14% for MM Baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MASHER instead of MM Base. | MultiHiEve event hierarchy prediction · MultiHiEve validation | Avg F1: 11.4 -> 18.0 | Helped |
| use MASHER instead of MM Base. | MultiHiEve event hierarchy prediction · MultiHiEve test | Avg F1: 11.4 -> 12.8 | Helped |
| add contextual video features (CT) (ablation) | MASHER · MultiHiEve validation | Avg F1: 14.9 -> 16.5 | Helped |
| add ConceptNet commonsense features (CS) (ablation) | MASHER · MultiHiEve validation | Avg F1: 14.9 -> 15.7 | Helped |
| add embedding interactions (EI) (ablation) | MASHER · MultiHiEve validation | Avg F1: 14.9 -> 16.3 | Helped |
| add CT, CS, and EI together (ablation) | MASHER · MultiHiEve validation | Avg F1: 14.9 -> 18.0 | Helped |
| use CT output rather than direct CLIP visual encoding as the visual input to CS (ablation) | MASHER · MultiHiEve validation | Avg F1: 17.1 -> 18.0 | Helped |
| use MASHER instead of MM Base. to retrieve visual-only subevents | MultiHiEve event hierarchy prediction · visual-only subevents | recall: 2.14% -> 15.92% | Helped |
| use predicted text events instead of ground-truth text events as MASHER input (ablation) | MASHER · MultiHiEve | Avg F1: 12.8 -> 7.8 | Hurt |
Surprising findings
- The generic Video-LLaMA zero-shot model underperforms the specialized MM Baseline pipeline, suggesting specialized models beat generic vision-language models here.
- Identical-relation precision is low for both MM Baseline and MASHER because video retrieval noisily labels hierarchical pairs as identical.
Limitations
- Low precision for identical relations due to noisy CLIP-based retrieval.
- Pseudo-label quality is limited; text-to-video retrieval precision is around 75% on manual check of 100 samples.
- Test annotation is small (526 pairs) and inter-annotator agreement is modest (47.5 hierarchical, 48.9 identical).
Experimental setup
Datasets: MultiHiEve
Benchmarks: MultiHiEve validation set MultiHiEve test set
Metrics: precision recall F1 average F1
Baselines: Prior Baseline Text Baseline (ASR-based) MM Baseline (pseudo-label pipeline) Video-LLaMA
Builds on: CLIP ConceptNet PySceneDetect Video-LLaMA
Closest papers by meaning
- MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval (CVPR 2025 · similarity 0.87)
- Cross-modal Multi-task Learning for Multimedia Event Extraction (AAAI 2025 · similarity 0.87)
- VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videos (AAAI 2025 · similarity 0.86)
- OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies (CVPR 2024 · similarity 0.85)
- EventGPT: Event Stream Understanding with Multimodal Large Language Models (CVPR 2025 · similarity 0.85)
- Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.