Research map / Video / Video-language understanding
VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the video event understanding task and the VidEvent dataset (1,110 movie recap videos, ~24k PropBank-style events, ~17.5k relations) with baselines for localization, extraction, relation classification, and script event induction.
Problem
Existing video understanding datasets capture atomic actions or captions and lack structured, hierarchical events with dynamic logical evolution; the paper targets extracting event scripts from videos and predicting next events from them.
Approach
Frames video event understanding as four linked subtasks (event localization, event structure extraction, relation classification, script event induction) grounded in PropBank event structures, MAVEN-ERE-style relations, and persistent person identifiers across a video, with a new dataset and baselines.
Key results
- Semantic aggregation on TriDet (TriDet+Agg) reaches 0.74 average mAP for event localization, versus 0.69 for TriDet with SlowFast features.
- Script event induction: text-only RoBERTa reaches 0.55 accuracy, and TF+RoBERTa multimodal reaches 0.56, while vision-only models reach 0.23-0.25.
- Event relation classification: SF+RoBERTa achieves the highest macro F1 of 0.48 among the tested models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic aggregation to TriDet with SlowFast (ablation) | TriDet (SlowFast) · VidEvent test | mAP: 0.69 -> 0.74 | Helped |
| Use joint rather than stepwise event extraction | Video event extraction · VidEvent test | CIDEr (event arguments): 0.39 -> 0.46 | Mixed |
| Use joint rather than stepwise event extraction | Video event extraction · VidEvent test | SPICE (event arguments): 0.25 -> 0.25 | No clear effect |
| Combine visual and text branches rather than use text alone (ablation) | Script event induction · VidEvent | accuracy: 0.55 -> 0.56 | Helped |
| Replace the text-only branch with the vision-only branch (ablation) | Script event induction · VidEvent | accuracy: 0.55 -> 0.23 | Hurt |
| Combine visual and text features | Event relation classification · VidEvent test | F1: 0.41 -> 0.48 | Helped |
Surprising findings
- Vision-only script event induction (0.23-0.25 accuracy) is far below text-only RoBERTa (0.55), and multimodal fusion adds only a slight gain.
Limitations
- The dataset provides only public URLs for videos rather than the video files, which may hinder reproducibility.
- Script event induction is simplified to a single-choice setting with five candidates.
- Baselines show large room for improvement, especially in visual-only reasoning.
Experimental setup
Datasets: VidEvent ActivityNet Captions MSR-VTT Vatex-en VidSitu
Benchmarks: VidEvent
Metrics: mAP@IoU P@5 R@5 F1@5 METEOR CIDEr SPICE top-1 accuracy macro precision macro recall
Baselines: ActionFormer TriDet SlowFast TimesFormer CLIP RoBERTa VisualBERT
Builds on: PropBank MAVEN-ERE ActionFormer TriDet CLIP
Closest papers by meaning
- Beyond Grounding: Extracting Fine-Grained Event Hierarchies across Modalities (AAAI 2024 · similarity 0.86)
- EventLens: Event-Structure Reinforcement Learning for Video Understanding (NeurIPS 2026 · similarity 0.86)
- Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras (NeurIPS 2025 · similarity 0.85)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.85)
- Streaming Detection of Queried Event Start (NeurIPS 2024 · similarity 0.85)
- EventZoom: A Progressive Approach to Event-Based Data Augmentation for Enhanced Neuromorphic Vision (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.